爬虫

浏览器环境检测

背景

之前爬虫 驱动个selenium基本上就可以了.

但是现在各种检测浏览器环境…特别是不熟悉js的同学就更烦了

本文是直接把 selenium pyppeteer 以及正常打开浏览器 的环境差异直接列出来

这样你就可以更愉快的爬虫了（可以直接把环境全部模拟上,或者大概看看有啥,下次看人家混淆js的时候心里有个数）

原理

就是遍历window对象,把属性全部保存成json文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
function recur(obj) {
    var result = {},
        _tmp;
    for (var i in obj) {
        // enabledPlugin is too nested, also skip functions
        if (i === 'enabledPlugin' || typeof obj[i] === 'function') {
            continue;
        } else if (typeof obj[i] === 'object') {
            // get props recursively
            _tmp = recur(obj[i]);
            // if object is not {}
            if (Object.keys(_tmp).length) {
                result[i] = _tmp;
            }
        } else {
            // string, number or boolean
            result[i] = obj[i];
        }
    }
    return result;
}

function go() {

    var j = new Object();
    var propertys = Object.getOwnPropertyNames(window);

    propertys.forEach(element => {
        if (element === 'globalThis' ||
            element === 'Illegal' ||
            element === 'parent' ||
            element === 'top' ||
            element === 'frames' ||
            element === 'self' ||
            element === 'window' ||
            element === 'document'
        ) {} else {
            eval("j." + element + "=recur(window." + element + ")")
        }
    });

    console.log(JSON.stringify(j))

}
go()

然后分别正常打开,selenium打开，pyppeteer打开

再查看方法的差异

操作

获取json

下载本项目

启动本地server python -m http.server 80

打开 http://localhost/

正常打开最好是无痕模式,因为浏览器扩展可能会导致多出很多浏览器扩展的属性

打开网页以后 -> 打开开发者工具 -> 切换到console

然后点击copy,粘贴json到对应的文件内

selenium pyppeteer 打开可以参考getEnvironment.py

比较

直接运行 python diff.py就可以了.

下面可以看到

差异有

root[‘navigator’][‘webdriver’], root[‘clientInformation’][‘webdriver’] 等等

这个算是目前检测用的比较多的,还有一些其他的可以自己运行看看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
selenuim 与正常环境的差异
############################################################
{ 'dictionary_item_added': [root['cdc_adoQpoasnfa76pfcZLmcfl_Symbol'], root['cdc_adoQpoasnfa76pfcZLmcfl_Array'], root['cdc_adoQpoasnfa76pfcZLmcfl_Promise'], root['navigator']['webdriver'], root['clientInformation']['webdriver']],
  'dictionary_item_removed': [root['chrome']['runtime'], root['navigator']['doNotTrack'], root['navigator']['languages']['2'], root['clientInformation']['doNotTrack'], root['clientInformation']['languages']['2']],
  'type_changes': { "root['visualViewport']['height']": { 'new_type': <class 'int'>,
                                                          'new_value': 948,
                                                          'old_type': <class 'float'>,
                                                          'old_value': 751.2000122070312}},
  'values_changed': { "root['clientInformation']['connection']['downlink']": { 'new_value': 1.3,
                                                                               'old_value': 1.45},
                      "root['screen']['availHeight']": { 'new_value': 1027,
                                                         'old_value': 1050},
                      "root['screen']['availTop']": { 'new_value': 23,
                                                      'old_value': 0},
                      "root['visualViewport']['width']": { 'new_value': 840,
                                                           'old_value': 1344}}}
pyppeteer 与正常环境的差异
############################################################
{ 'dictionary_item_added': [root['WebKitAnimationEvent'], root['RTCRtpContributingSource'], root['SVGDiscardElement'], root['MediaCapabilitiesInfo'], root['WebKitTransitionEvent'], root['chrome']['webstore'], root['navigator']['webdriver'], root['clientInformation']['webdriver']],
  'dictionary_item_removed': [root['BackgroundFetchManager'], root['RTCPeerConnectionIceErrorEvent'], root['WritableStreamDefaultWriter'], root['XRDOMOverlayState'], root['TrustedTypePolicy'], root['GeolocationCoordinates'], root['External'], root['ClipboardItem'], root['XRTransientInputHitTestSource'], root['XRRay'], root['PerformanceElementTiming'], root['XRViewerPose'], root['RTCError'], root['TextDecoderStream'], root['BackgroundFetchRegistration'], root['TrustedHTML'], root['onpointerrawupdate'], root['XRSession'], root['XRSessionEvent'], root['XRReferenceSpaceEvent'], root['XRReferenceSpace'], root['BarcodeDetector'], root['SubmitEvent'], root['AnimationEffect'], root['PeriodicSyncManager'], root['onformdata'], root['onselectstart'], root['TrustedScriptURL'], root['XRRenderState'], root['SpeechSynthesisErrorEvent'], root['RTCIceTransport'], root['TextEncoderStream'], root['MediaSession'], root['XRHitTestSource'], root['GeolocationPositionError'], root['XRPose'], root['XRView'], root['ElementInternals'], root['XRInputSourceArray'], root['XRInputSourceEvent'], root['XRFrame'], root['DecompressionStream'], root['TrustedScript'], root['Geolocation'], root['TrustedTypePolicyFactory'], root['XRBoundedReferenceSpace'], root['FeaturePolicy'], root['PerformanceEventTiming'], root['LargestContentfulPaint'], root['VideoPlaybackQuality'], root['onselectionchange'], root['XRSpace'], root['XRInputSourcesChangeEvent'], root['GeolocationPosition'], root['XRViewport'], root['XRHitTestResult'], root['queueMicrotask'], root['XRWebGLLayer'], root['DeviceMotionEventRotationRate'], root['ReadableStreamDefaultReader'], root['BackgroundFetchRecord'], root['LayoutShift'], root['DeviceMotionEventAcceleration'], root['MediaMetadata'], root['XRTransientInputHitTestResult'], root['PaymentMethodChangeEvent'], root['trustedTypes'], root['FragmentDirective'], root['UserActivation'], root['RTCErrorEvent'], root['CompressionStream'], root['XRRigidTransform'], root['Animation'], root['RTCSctpTransport'], root['XRInputSource'], root['FormDataEvent'], root['XRSystem'], root['EnterPictureInPictureEvent'], root['KeyframeEffect'], root['RTCDtlsTransport'], root['chrome']['runtime'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_OVERLAY'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_HUE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_SATURATION'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR_DODGE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_LUMINOSITY'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_EXCLUSION'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_DIFFERENCE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_SOFT_LIGHT'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_HARD_LIGHT'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR_BURN'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR'], root['navigator']['userActivation'], root['navigator']['doNotTrack'], root['navigator']['mediaSession'], root['navigator']['languages']['2'], root['clientInformation']['userActivation'], root['clientInformation']['doNotTrack'], root['clientInformation']['mediaSession'], root['clientInformation']['languages']['2'], root['PerformanceObserver']['supportedEntryTypes']],
  'type_changes': { "root['visualViewport']['height']": { 'new_type': <class 'int'>,
                                                          'new_value': 600,
                                                          'old_type': <class 'float'>,
                                                          'old_value': 751.2000122070312}},
  'values_changed': { "root['clientInformation']['appVersion']": { 'new_value': '5.0 '
                                                                                '(Macintosh; '
                                                                                'Intel '
                                                                                'Mac '
                                                                                'OS '
                                                                                'X '
                                                                                '10_15_4) '
                                                                                'AppleWebKit/537.36 '
                                                                                '(KHTML, '
                                                                                'like '
                                                                                'Gecko) '
                                                                                'Chrome/71.0.3542.0 '
                                                                                'Safari/537.36',
                                                                   'old_value': '5.0 '
                                                                                '(Macintosh; '
                                                                                'Intel '
                                                                                'Mac '
                                                                                'OS '
                                                                                'X '
                                                                                '10_15_4) '
                                                                                'AppleWebKit/537.36 '
                                                                                '(KHTML, '
                                                                                'like '
                                                                                'Gecko) '
                                                                                'Chrome/83.0.4103.61 '
                                                                                'Safari/537.36'},
                      "root['clientInformation']['connection']['rtt']": { 'new_value': 100,
                                                                          'old_value': 150},
                      "root['clientInformation']['plugins']['0']['name']": { 'new_value': 'Chromium '
                                                                                          'PDF '
                                                                                          'Plugin',
                                                                             'old_value': 'Chrome '
                                                                                          'PDF '
                                                                                          'Plugin'},
                      "root['clientInformation']['plugins']['1']['name']": { 'new_value': 'Chromium '
                                                                                          'PDF '
                                                                                          'Viewer',
                                                                             'old_value': 'Chrome '
                                                                                          'PDF '
                                                                                          'Viewer'},
                      "root['clientInformation']['userAgent']": { 'new_value': 'Mozilla/5.0 '
                                                                               '(Macintosh; '
                                                                               'Intel '
                                                                               'Mac '
                                                                               'OS '
                                                                               'X '
                                                                               '10_15_4) '
                                                                               'AppleWebKit/537.36 '
                                                                               '(KHTML, '
                                                                               'like '
                                                                               'Gecko) '
                                                                               'Chrome/71.0.3542.0 '
                                                                               'Safari/537.36',
                                                                  'old_value': 'Mozilla/5.0 '
                                                                               '(Macintosh; '
                                                                               'Intel '
                                                                               'Mac '
                                                                               'OS '
                                                                               'X '
                                                                               '10_15_4) '
                                                                               'AppleWebKit/537.36 '
                                                                               '(KHTML, '
                                                                               'like '
                                                                               'Gecko) '
                                                                               'Chrome/83.0.4103.61 '
                                                                               'Safari/537.36'},
                      "root['console']['memory']['jsHeapSizeLimit']": { 'new_value': 2330000000,
                                                                        'old_value': 3760000000},
                      "root['history']['length']": { 'new_value': 2,
                                                     'old_value': 3},
                      "root['location']['href']": { 'new_value': 'http://localhost/',
                                                    'old_value': 'http://localhost/t.html'},
                      "root['location']['pathname']": { 'new_value': '/',
                                                        'old_value': '/t.html'},
                      "root['navigator']['appVersion']": { 'new_value': '5.0 '
                                                                        '(Macintosh; '
                                                                        'Intel '
                                                                        'Mac '
                                                                        'OS X '
                                                                        '10_15_4) '
                                                                        'AppleWebKit/537.36 '
                                                                        '(KHTML, '
                                                                        'like '
                                                                        'Gecko) '
                                                                        'Chrome/71.0.3542.0 '
                                                                        'Safari/537.36',
                                                           'old_value': '5.0 '
                                                                        '(Macintosh; '
                                                                        'Intel '
                                                                        'Mac '
                                                                        'OS X '
                                                                        '10_15_4) '
                                                                        'AppleWebKit/537.36 '
                                                                        '(KHTML, '
                                                                        'like '
                                                                        'Gecko) '
                                                                        'Chrome/83.0.4103.61 '
                                                                        'Safari/537.36'},
                      "root['navigator']['connection']['rtt']": { 'new_value': 100,
                                                                  'old_value': 150},
                      "root['navigator']['plugins']['0']['name']": { 'new_value': 'Chromium '
                                                                                  'PDF '
                                                                                  'Plugin',
                                                                     'old_value': 'Chrome '
                                                                                  'PDF '
                                                                                  'Plugin'},
                      "root['navigator']['plugins']['1']['name']": { 'new_value': 'Chromium '
                                                                                  'PDF '
                                                                                  'Viewer',
                                                                     'old_value': 'Chrome '
                                                                                  'PDF '
                                                                                  'Viewer'},
                      "root['screen']['availHeight']": { 'new_value': 1027,
                                                         'old_value': 1050},
                      "root['screen']['availTop']": { 'new_value': 23,
                                                      'old_value': 0},
                      "root['screen']['colorDepth']": { 'new_value': 24,
                                                        'old_value': 30},
                      "root['screen']['orientation']['type']": { 'new_value': 'portrait-primary',
                                                                 'old_value': 'landscape-primary'},
                      "root['screen']['pixelDepth']": { 'new_value': 24,
                                                        'old_value': 30},
                      "root['visualViewport']['width']": { 'new_value': 800,
                                                           'old_value': 1344}}}

项目地址

https://github.com/jin10086/browser-environment-test

推荐文章