Python 进行查询日志查询条件分析

任务：crm日志的查询条件每次是哪几个字段查，有几种组合，统计每种组合查询的量

日志样例：

 132.xxx.xx.x -  -  [-- ::] "GET /REST/HTableService?appId=crmyun&partition=2017&query=QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData&latnId=755&staffNo=GZTEST200&timestamp=1506585708188&signature=D73E9B59E08EA7B1C2D0DDA72AC957E4 HTTP/1.1"

 132.xxx.xx.x -  -  [-- ::] "GET /REST/HTableService?staffNo=xxTEST200&appId=crmyun&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData&timestamp=1505871359000&signature=6743AE272C10BCC2261E11AF4CA5EA19&charset=UTF-8&partition=2017&query=STAFF_ID=1212100141|and|DATETIME>20170917000000000|and|DATETIME<20170919000000000 HTTP/1.1"

查询条件：query查询条件可以多个，用|and|分割。

步骤：

1、正则获取query查询条件组合

 query=QUERY_TYPE1%3D%E6%8E%A5%E5%%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000

 query=STAFF_ID=|and|DATETIME>|and|DATETIME<

2、截取query列表，得到查询条件组合；以"%7C|\|"分割，得到列表

 ['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'and', 'QUERY_VALUE1%3D17727955834', 'and', 'DATETIME%3E20170925000000000', 'and', 'DATETIME%3C20170928000000000']

 ['STAFF_ID=1212100141', 'and', 'DATETIME>20170917000000000', 'and', 'DATETIME<20170919000000000']

3、剔除 'and' 项(列表取[::2])得到新列表

 ['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'QUERY_VALUE1%3D17727955834', 'DATETIME%3E20170925000000000', 'DATETIME%3C20170928000000000']

 ['STAFF_ID=1212100141', 'DATETIME>20170917000000000', 'DATETIME<20170919000000000']

4、以'%3D|%3E|%3C|>|<|='分割，并将key放入set()中，得到去重后的结果

 ['QUERY_TYPE1', '%E6%8E%A5%E5%85%A5%E5%8F%B7']

 ['QUERY_VALUE1', '']

 ['DATETIME', '']

 ['DATETIME', '']

 ['STAFF_ID', '']

 ['DATETIME', '']

 ['DATETIME', '']

5、将列表key值放入set()中，得到结果；参考代码如下

 import sys

 import time

 import re

 def read_write():

     with open("C:\\Users\\admin\\Desktop\\c5.log", 'r') as f1:

         for line in f1.readlines():

             pattern = re.compile(r'query=.*?\s')

             results = re.search(pattern, line).group().split('&')

             for result in results:

                 if result.startswith("query"):

                     temp = result[6: ]

                     list = re.split("%7C|\|",temp)[::2]

 #                    print list

                     my_set = set()

                     for l in list:

                         arrya = re.split('%3D|%3E|%3C|>|<|=', l)

                         my_set.add(arrya[0])

                     print my_set

                     c = [i for i in my_set]

                     file = open("C:\\Users\\admin\\Desktop\\4.txt", 'a')

                     file.write(repr(c)+'\n')

                     file.close()

 if __name__ == '__main__':

     start = time.time()

     read_write()

     stop = time.time()

     print "running time is "+str(stop - start)

巴特西

Python 进行查询日志查询条件分析

最新文章

热门文章