浏览器环境检测

背景

之前爬虫 驱动个selenium基本上就可以了.

但是现在各种检测浏览器环境…特别是不熟悉js的同学就更烦了

本文是直接把 selenium pyppeteer 以及正常打开浏览器 的环境差异直接列出来

这样你就可以更愉快的爬虫了(可以直接把环境全部模拟上,或者大概看看有啥,下次看人家混淆js的时候心里有个数)

原理

就是遍历window对象,把属性全部保存成json文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
function recur(obj) {
var result = {},
_tmp;
for (var i in obj) {
// enabledPlugin is too nested, also skip functions
if (i === 'enabledPlugin' || typeof obj[i] === 'function') {
continue;
} else if (typeof obj[i] === 'object') {
// get props recursively
_tmp = recur(obj[i]);
// if object is not {}
if (Object.keys(_tmp).length) {
result[i] = _tmp;
}
} else {
// string, number or boolean
result[i] = obj[i];
}
}
return result;
}

function go() {

var j = new Object();
var propertys = Object.getOwnPropertyNames(window);

propertys.forEach(element => {
if (element === 'globalThis' ||
element === 'Illegal' ||
element === 'parent' ||
element === 'top' ||
element === 'frames' ||
element === 'self' ||
element === 'window' ||
element === 'document'
) {} else {
eval("j." + element + "=recur(window." + element + ")")
}
});

console.log(JSON.stringify(j))

}
go()

然后分别正常打开,selenium打开,pyppeteer打开

再查看方法的差异

操作

获取json

下载本项目

启动本地server python -m http.server 80

打开 http://localhost/

正常打开最好是无痕模式,因为浏览器扩展可能会导致多出很多浏览器扩展的属性

打开网页以后 -> 打开开发者工具 -> 切换到console

然后点击copy,粘贴json到对应的文件内

selenium pyppeteer 打开可以参考getEnvironment.py

比较

直接运行 python diff.py就可以了.

下面可以看到

差异有

root[‘navigator’][‘webdriver’], root[‘clientInformation’][‘webdriver’] 等等

这个算是目前检测用的比较多的,还有一些其他的可以自己运行看看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
selenuim 与正常环境的差异
############################################################
{ 'dictionary_item_added': [root['cdc_adoQpoasnfa76pfcZLmcfl_Symbol'], root['cdc_adoQpoasnfa76pfcZLmcfl_Array'], root['cdc_adoQpoasnfa76pfcZLmcfl_Promise'], root['navigator']['webdriver'], root['clientInformation']['webdriver']],
'dictionary_item_removed': [root['chrome']['runtime'], root['navigator']['doNotTrack'], root['navigator']['languages']['2'], root['clientInformation']['doNotTrack'], root['clientInformation']['languages']['2']],
'type_changes': { "root['visualViewport']['height']": { 'new_type': <class 'int'>,
'new_value': 948,
'old_type': <class 'float'>,
'old_value': 751.2000122070312}},
'values_changed': { "root['clientInformation']['connection']['downlink']": { 'new_value': 1.3,
'old_value': 1.45},
"root['screen']['availHeight']": { 'new_value': 1027,
'old_value': 1050},
"root['screen']['availTop']": { 'new_value': 23,
'old_value': 0},
"root['visualViewport']['width']": { 'new_value': 840,
'old_value': 1344}}}
pyppeteer 与正常环境的差异
############################################################
{ 'dictionary_item_added': [root['WebKitAnimationEvent'], root['RTCRtpContributingSource'], root['SVGDiscardElement'], root['MediaCapabilitiesInfo'], root['WebKitTransitionEvent'], root['chrome']['webstore'], root['navigator']['webdriver'], root['clientInformation']['webdriver']],
'dictionary_item_removed': [root['BackgroundFetchManager'], root['RTCPeerConnectionIceErrorEvent'], root['WritableStreamDefaultWriter'], root['XRDOMOverlayState'], root['TrustedTypePolicy'], root['GeolocationCoordinates'], root['External'], root['ClipboardItem'], root['XRTransientInputHitTestSource'], root['XRRay'], root['PerformanceElementTiming'], root['XRViewerPose'], root['RTCError'], root['TextDecoderStream'], root['BackgroundFetchRegistration'], root['TrustedHTML'], root['onpointerrawupdate'], root['XRSession'], root['XRSessionEvent'], root['XRReferenceSpaceEvent'], root['XRReferenceSpace'], root['BarcodeDetector'], root['SubmitEvent'], root['AnimationEffect'], root['PeriodicSyncManager'], root['onformdata'], root['onselectstart'], root['TrustedScriptURL'], root['XRRenderState'], root['SpeechSynthesisErrorEvent'], root['RTCIceTransport'], root['TextEncoderStream'], root['MediaSession'], root['XRHitTestSource'], root['GeolocationPositionError'], root['XRPose'], root['XRView'], root['ElementInternals'], root['XRInputSourceArray'], root['XRInputSourceEvent'], root['XRFrame'], root['DecompressionStream'], root['TrustedScript'], root['Geolocation'], root['TrustedTypePolicyFactory'], root['XRBoundedReferenceSpace'], root['FeaturePolicy'], root['PerformanceEventTiming'], root['LargestContentfulPaint'], root['VideoPlaybackQuality'], root['onselectionchange'], root['XRSpace'], root['XRInputSourcesChangeEvent'], root['GeolocationPosition'], root['XRViewport'], root['XRHitTestResult'], root['queueMicrotask'], root['XRWebGLLayer'], root['DeviceMotionEventRotationRate'], root['ReadableStreamDefaultReader'], root['BackgroundFetchRecord'], root['LayoutShift'], root['DeviceMotionEventAcceleration'], root['MediaMetadata'], root['XRTransientInputHitTestResult'], root['PaymentMethodChangeEvent'], root['trustedTypes'], root['FragmentDirective'], root['UserActivation'], root['RTCErrorEvent'], root['CompressionStream'], root['XRRigidTransform'], root['Animation'], root['RTCSctpTransport'], root['XRInputSource'], root['FormDataEvent'], root['XRSystem'], root['EnterPictureInPictureEvent'], root['KeyframeEffect'], root['RTCDtlsTransport'], root['chrome']['runtime'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_OVERLAY'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_HUE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_SATURATION'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR_DODGE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_LUMINOSITY'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_EXCLUSION'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_DIFFERENCE'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_SOFT_LIGHT'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_HARD_LIGHT'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR_BURN'], root['SVGFEBlendElement']['SVG_FEBLEND_MODE_COLOR'], root['navigator']['userActivation'], root['navigator']['doNotTrack'], root['navigator']['mediaSession'], root['navigator']['languages']['2'], root['clientInformation']['userActivation'], root['clientInformation']['doNotTrack'], root['clientInformation']['mediaSession'], root['clientInformation']['languages']['2'], root['PerformanceObserver']['supportedEntryTypes']],
'type_changes': { "root['visualViewport']['height']": { 'new_type': <class 'int'>,
'new_value': 600,
'old_type': <class 'float'>,
'old_value': 751.2000122070312}},
'values_changed': { "root['clientInformation']['appVersion']": { 'new_value': '5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS '
'X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/71.0.3542.0 '
'Safari/537.36',
'old_value': '5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS '
'X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/83.0.4103.61 '
'Safari/537.36'},
"root['clientInformation']['connection']['rtt']": { 'new_value': 100,
'old_value': 150},
"root['clientInformation']['plugins']['0']['name']": { 'new_value': 'Chromium '
'PDF '
'Plugin',
'old_value': 'Chrome '
'PDF '
'Plugin'},
"root['clientInformation']['plugins']['1']['name']": { 'new_value': 'Chromium '
'PDF '
'Viewer',
'old_value': 'Chrome '
'PDF '
'Viewer'},
"root['clientInformation']['userAgent']": { 'new_value': 'Mozilla/5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS '
'X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/71.0.3542.0 '
'Safari/537.36',
'old_value': 'Mozilla/5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS '
'X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/83.0.4103.61 '
'Safari/537.36'},
"root['console']['memory']['jsHeapSizeLimit']": { 'new_value': 2330000000,
'old_value': 3760000000},
"root['history']['length']": { 'new_value': 2,
'old_value': 3},
"root['location']['href']": { 'new_value': 'http://localhost/',
'old_value': 'http://localhost/t.html'},
"root['location']['pathname']": { 'new_value': '/',
'old_value': '/t.html'},
"root['navigator']['appVersion']": { 'new_value': '5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/71.0.3542.0 '
'Safari/537.36',
'old_value': '5.0 '
'(Macintosh; '
'Intel '
'Mac '
'OS X '
'10_15_4) '
'AppleWebKit/537.36 '
'(KHTML, '
'like '
'Gecko) '
'Chrome/83.0.4103.61 '
'Safari/537.36'},
"root['navigator']['connection']['rtt']": { 'new_value': 100,
'old_value': 150},
"root['navigator']['plugins']['0']['name']": { 'new_value': 'Chromium '
'PDF '
'Plugin',
'old_value': 'Chrome '
'PDF '
'Plugin'},
"root['navigator']['plugins']['1']['name']": { 'new_value': 'Chromium '
'PDF '
'Viewer',
'old_value': 'Chrome '
'PDF '
'Viewer'},
"root['screen']['availHeight']": { 'new_value': 1027,
'old_value': 1050},
"root['screen']['availTop']": { 'new_value': 23,
'old_value': 0},
"root['screen']['colorDepth']": { 'new_value': 24,
'old_value': 30},
"root['screen']['orientation']['type']": { 'new_value': 'portrait-primary',
'old_value': 'landscape-primary'},
"root['screen']['pixelDepth']": { 'new_value': 24,
'old_value': 30},
"root['visualViewport']['width']": { 'new_value': 800,
'old_value': 1344}}}

项目地址

https://github.com/jin10086/browser-environment-test


本文作者:高金
本文地址https://igaojin.me/2020/05/23/浏览器环境检测/
版权声明:转载请注明出处!

推荐文章