squid3.3.8

megahertz

Posted April 7, 2015 (edited) · Report post

Добрый вечер форумчане

Есть кто-то кто настраивал прозрачный кэш прокси на squid v3.3.8 ?

Есть Ubuntu 14 на котором развернут squid 3.3.8

Сейчас при текущей конфигурации:

acl localnet src 10.20.30.0/24
http_access allow all
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#http_access deny !Safe_ports
#http_access deny CONNECT !SSL_ports
#http_access allow localhost manager
#http_access allow manager
#http_access allow localhost
#http_access allow localnet
#http_access deny all

#dns_v4_first on
dns_nameservers 8.8.8.8
visible_hostname mynetwork
http_port 10.20.30.254:3128 intercept (пробовал делать transparent)
http_port 10.20.30.254:3129

http_reply_access allow all

coredump_dir /home/mhz/cache
cache_dir ufs /home/mhz/cache 100 16 256
access_log /home/mhz/access.log

в access.log следующее:

1428431950.118      0 10.20.30.254 TCP_MISS/403 4042 GET http://www.squid-cache.org/Artwork/SN.png - HIER_NONE/- text/html
1428431950.119      1 10.20.30.245 TCP_MISS/403 4137 GET http://www.squid-cache.org/Artwork/SN.png - HIER_DIRECT/10.20.30.254 text/html

в cache.log:

2015/04/07 23:39:51| WARNING: Error Pages Missing Language: en-ru
2015/04/07 23:39:51| WARNING: Forwarding loop detected for:
GET /Artwork/SN.png HTTP/1.1
Host: www.squid-cache.org
Accept: image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Linux; Android 4.4.3; HTC One dual sim Build/KTU84L) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36
Referer: http://www.e1.ru/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-RU,en-US;q=0.8,en;q=0.6
Via: 1.1 network (squid/3.3.8)
X-Forwarded-For: 10.20.30.245
Cache-Control: max-age=259200
Connection: keep-alive

по схеме users(10.20.30.0/24) --- mikrotik(10.20.30.1) --- squid3(10.20.30.254 прозрачный) --- internet

iptables на убунте со squid3 совершенно пуст т.к форвардинг на порт 3128 делается микротиком.

Собственно исходя из всего этого вопрос, где недоглядел?

Edited April 7, 2015 by megahertz

Share this post

Link to post

Share on other sites

Ivan_83

Posted April 7, 2015 · Report post

#http_access allow localnet

Раскомментируй, и вообще, конфиг какой то покоцанный.

А за 8.8.8.8 дай себе молотком по...любой части тела.

Share this post

Link to post

Share on other sites

megahertz

Posted April 8, 2015 · Report post

#http_access allow localnet

Раскомментируй, и вообще, конфиг какой то покоцанный.

А за 8.8.8.8 дай себе молотком по...любой части тела.

на самом деле это уже все эксперементы в прцоессе запуска, но тем не менее убрал и получил:

access.log - чист, ничего не сыпется

cache.log - NF getsockopt(SO_ORIGINAL_DST) failed on local=10.20.30.254:3128 remote=10.20.30.245:37071 FD 10 flags=33: (92

) Protocol not available

Share this post

Link to post

Share on other sites

[anp/hsw]

Posted April 8, 2015 · Report post

NF getsockopt(SO_ORIGINAL_DST) failed on local=10.20.30.254:3128 remote=10.20.30.245:37071 FD 10 flags=33: (92

) Protocol not available

Он у вас адрес подделать не может. Смотрите в сторону опций ip_nonlocal_bind в sysctl/ядре.

Share this post

Link to post

Share on other sites

Ivan_83

Posted April 9, 2015 · Report post

WARNING: Error Pages Missing Language: en-ru - что не понятно то?

Или пропиши в настройках скивда правильный язык или положи страничики с ошибками локализованные куда надо.

Что с е1 - хз, толи сам сайт по кругу редиректит толи что то ещё в настройках.

А вот пользовательские куки выкладывать в паблик не хорошо, разве что они твои и тебе не жалко :)

Share this post

Link to post

Share on other sites

[anp/hsw]

Posted April 10, 2015 · Report post

iptables на убунте со squid3 совершенно пуст т.к форвардинг на порт 3128 делается микротиком.

WARNING: Forwarding loop detected for:

И кто за вас эту петлю будет убирать?

Читайте https://www.kernel.org/doc/Documentation/networking/tproxy.txt

Share this post

Link to post

Share on other sites

megahertz

Posted April 11, 2015 · Report post

Спасибо, попробую и по результатам отпишусь

Share this post

Link to post

Share on other sites

Ivan_83

Posted April 12, 2015 · Report post

Поставь лайтсквид для анализа логов.

Он умеет считать процент попадания в кеш, вот 10% уже нормально, 20% - супер.

Share this post

Link to post

Share on other sites

npokypop

Posted April 14, 2015 · Report post

Все больше ресурсов отдают свой контент по https, его все же без nginx трудно как-то кэшировать, я про соц. сети в частности.

Share this post

Link to post

Share on other sites

megahertz

Posted April 16, 2015 (edited) · Report post

Да, но по сути что мешает кэшировать *.jpg,*.jpeg, *.png и т.п? интерес как раз в том чтобы кэшировать только эти ресурсы, при большом количестве пользователей повторное обращение к тем же ресурсам существенно снизят нагрузку на аплинк, сейчас у меня squid3.3.8 вообще ничего не кэширует, в логах светятся одни TCP_MISS/200

с вашего совета смотрю в сторону nginx

Edited April 16, 2015 by megahertz

Share this post

Link to post

Share on other sites

npokypop

Posted April 16, 2015 (edited) · Report post

Связка squid+nginx, если лень изначально самому разбираться с конфигами, то советую https://aacable.wordpress.com/2014/07/02/lusca-automated-install-script , работает из коробки.

Edited April 16, 2015 by npokypop

Share this post

Link to post

Share on other sites

megahertz

Posted April 17, 2015 · Report post

да, lusca завелся сходу, без шаманства, прозрачно работает но все тот же трабл с кэшированием

1429261692.131 285 10.20.30.245 TCP_MISS/302 457 GET http://habrastorage.org/files/623/d37/d81/623d37d812404ce4ad5e67c70cd9ec5a.jpg - DIRECT

/82.192.95.170 text/html

1429261692.280 253 10.20.30.245 TCP_MISS/302 457 GET http://habrastorage.org/files/482/1a8/146/4821a81465b448da935039214674ebd0.jpg - DIRECT

/82.192.95.170 text/html

1429261693.199 518 10.20.30.245 TCP_MISS/200 49010 GET http://hsto.org/files/623/d37/d81/623d37d812404ce4ad5e67c70cd9ec5a.jpg - DIRECT/108.1

62.207.55 image/jpeg

1429261693.394 712 10.20.30.245 TCP_MISS/200 60318 GET http://hsto.org/files/482/1a8/146/4821a81465b448da935039214674ebd0.jpg -

видно что ничего не закэшировалось

конфиг:

http_port 3128 transparent
server_http11 on
pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid/
error_directory /usr/share/squid/errors/English
icon_directory /usr/share/squid/icons
mime_table /etc/squid/mime.conf
access_log daemon:/var/log/squid/access.log squid
cache_log none
referer_log /var/log/squid/referer.log
cache_store_log none
store_dir_select_algorithm  round-robin
logfile_daemon /usr/lib/squid/logfile-daemon
logfile_rotate 1
cache_mem 256 MB
maximum_object_size_in_memory 0 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 1 GB
cache_swap_low 98
cache_swap_high 99
cache_dir aufs /home/mhz/cache 5000 16 256
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl localnet src 10.0.0.0/8            # RFC1918 possible internal network
acl localnet src 172.16.0.0/12        # RFC1918 possible internal network
acl localnet src 192.168.0.0/16        # RFC1918 possible internal network
acl localnet src 125.165.92.1        # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80                # http
acl Safe_ports port 21                # ftp
acl Safe_ports port 443                # https
acl Safe_ports port 70                # gopher
acl Safe_ports port 210                # wais
acl Safe_ports port 1025-65535        # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488                # gss-http
acl Safe_ports port 591                # filemaker
acl Safe_ports port 777                # multiling http
acl CONNECT method CONNECT
acl purge method PURGE
acl snmppublic snmp_community public
acl range dstdomain .windowsupdate.com
range_offset_limit -1 KB range
acl DENYCACHE urlpath_regex \.(ini|ui|lst|inf|pak|ver|patch|md5|cfg|lst|list|rsc|log|conf|dbd|db)$
acl DENYCACHE urlpath_regex (notice.html|afs.dat|dat.asp|patchinfo.xml|version.list|iepngfix.htc|updates.txt|patchlist.txt)
acl DENYCACHE urlpath_regex (pointblank.css|login_form.css|form.css|noupdate.ui|ahn.ui|3n.mh)$
acl DENYCACHE urlpath_regex (Loader|gamenotice|sources|captcha|notice|reset)
no_cache deny DENYCACHE
range_offset_limit 1 MB !DENYCACHE
uri_whitespace strip
acl ads url_regex -i .youtube\.com\/ad_frame?
acl ads url_regex -i .(s|s[0-90-9])\.youtube\.com
acl ads url_regex -i .googlesyndication\.com
acl ads url_regex -i .doubleclick\.net
acl ads url_regex -i ^http:\/\/googleads\.*
acl ads url_regex -i ^http:\/\/(ad|ads|ads[0-90-9]|ads\d|kad|a[b|d]|ad\d|adserver|adsbox)\.[a-z0-9]*\.[a-z][a-z]*
acl ads url_regex -i ^http:\/\/openx\.[a-z0-9]*\.[a-z][a-z]*
acl ads url_regex -i ^http:\/\/[a-z0-9]*\.openx\.net\/
acl ads url_regex -i ^http:\/\/[a-z0-9]*\.u-ad\.info\/
acl ads url_regex -i ^http:\/\/adserver\.bs\/
acl ads url_regex -i !^http:\/\/adf\.ly
http_access deny ads
http_reply_access deny ads
strip_query_terms off
acl yutub url_regex -i .*youtube\.com\/.*$
acl yutub url_regex -i .*youtu\.be\/.*$
logformat squid1 %{Referer}>h %ru
access_log /var/log/squid/yt.log squid1 yutub
acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback.*id)
acl store_rewrite_list urlpath_regex \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)\?
acl store_rewrite_list urlpath_regex \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)\?
acl store_rewrite_list urlpath_regex \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)\?
acl store_rewrite_list urlpath_regex \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)\?
acl store_rewrite_list urlpath_regex \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)\?
acl store_rewrite_list urlpath_regex \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)\?
acl store_rewrite_list urlpath_regex \.(htm|html|mhtml|css|js)\?
acl store_rewrite_list_web url_regex ^http:\/\/([A-Za-z-]+[0-9]+)*\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_web_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.com doubleclick\.net
acl store_rewrite_list_path urlpath_regex \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)$
acl store_rewrite_list_path urlpath_regex \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)$
acl store_rewrite_list_path urlpath_regex \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)$
acl store_rewrite_list_path urlpath_regex \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)$
acl store_rewrite_list_path urlpath_regex \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)$
acl store_rewrite_list_path urlpath_regex \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)$
acl store_rewrite_list_path urlpath_regex \.(htm|html|mhtml|css|js)$
acl getmethod method GET
storeurl_access deny !getmethod
storeurl_access allow store_rewrite_list_web_CDN
storeurl_access allow store_rewrite_list_web store_rewrite_list_path
storeurl_access allow store_rewrite_list
storeurl_access deny all
storeurl_rewrite_program /etc/squid/storeurl.pl
storeurl_rewrite_children 10
storeurl_rewrite_concurrency 40
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv\?|\.fid\?) 43200 99% 43200 override-expire ignore-reload ignore-must-revalidate ignore-private
refresh_pattern -i (get_video\?|videoplayback\?|videodownload\?) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern ^http://*.jobstreet.com.*/.* 720 100% 10080 override-expire override-lastmod ignore-no-cache
refresh_pattern ^http://*.indowebster.com.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-auth
refresh_pattern ^http://*.21cineplex.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-auth
refresh_pattern ^http://*.atmajaya.*/.* 720 100% 10080 override-expire ignore-no-cache ignore-auth
refresh_pattern ^http://*.kompas.*/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.theinquirer.*/.* 720 100% 10080 override-expire ignore-no-cache ignore-auth
refresh_pattern ^http://*.blogspot.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.wordpress.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache
refresh_pattern ^http://*.photobucket.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.tinypic.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.imageshack.us/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.kaskus.*/.* 720 100% 28800 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://www.kaskus.com/.* 720 100% 28800 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.detik.*/.* 720 50% 2880 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.detiknews.*/*.* 720 50% 2880 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://video.liputan6.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://static.liputan6.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.friendster.com/.* 720 100% 10080 override-expire override-lastmod ignore-no-cache ignore-auth
refresh_pattern ^http://*.facebook.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://apps.facebook.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.fbcdn.net/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://profile.ak.fbcdn.net/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://static.playspoon.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://cooking.game.playspoon.com/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern -i http://[^a-z\.]*onemanga\.com/? 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://media?.onemanga.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.yahoo.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.google.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.forummikrotik.com/.* 720 80% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern ^http://*.linux.or.id/.* 720 100% 10080 override-expire override-lastmod reload-into-ims ignore-no-cache ignore-auth
refresh_pattern -i \.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)(\?.*|$) 5259487 999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache ignore-private
refresh_pattern -i \.(htm|html|mhtml|css|js)(\?.*|$) 1440 90% 86400 override-expire ignore-reload reload-into-ims
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern ^ftp: 10080 95% 10080 override-lastmod reload-into-ims
refresh_pattern . 0 20% 10080 override-lastmod reload-into-ims
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow all
http_access deny all
icp_access allow localnet
icp_access deny all
icp_port 0
buffered_logs on
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
forwarded_for off
header_access From deny all
header_access Server deny all
header_access Link deny all
header_access Via deny all
header_access X-Forwarded-For deny all
httpd_suppress_version_string on
shutdown_lifetime 10 seconds
snmp_port 3401
snmp_access allow snmppublic all
dns_timeout 1 minutes
dns_nameservers 8.8.8.8 8.8.4.4
fqdncache_size 5000    #16384
ipcache_size 5000    #16384
ipcache_low 98
ipcache_high 99
log_fqdn off
log_icp_queries off
memory_pools off
maximum_single_addr_tries 2
retry_on_error on
icp_hit_stale on
strip_query_terms off
query_icmp on
reload_into_ims on
emulate_httpd_log off
negative_ttl 0 seconds
pipeline_prefetch on
vary_ignore_expire on
half_closed_clients off
high_page_fault_warning 2
nonhierarchical_direct on
prefer_direct off
cache_mgr aacable@hotmail.com
cache_effective_user proxy
cache_effective_group proxy
visible_hostname proxy.zaib
unique_hostname syed_jahanzaib
cachemgr_passwd none all
client_db on
max_filedescriptors 8192
zph_mode tos
zph_local 0x30
zph_parent 0
zph_option 136

Share this post

Link to post

Share on other sites

megahertz

Posted April 17, 2015 (edited) · Report post

TCP_IMS_HIT304 325 GET http://cdn.clicktale.net/www/tc/WRe21.js - NONE/- application/javascript
TCP_IMS_HIT/304 325 GET http://cdn.clicktale.net/www/ChangeMonitor-latest.js - NONE/- application/javascript
TCP_MEM_HIT/200 347 GET http://www.msftconnecttest.com/connecttest.txt - NONE/- text/plain

а вот JS пролез

Edited April 17, 2015 by megahertz

Share this post

Link to post

Share on other sites

npokypop

Posted April 20, 2015 · Report post

lusca у меня хорошо кэширует такие ресурсы как megogo и аналоги, youtube кэшировать так я ее и не заставил.

Share this post

Link to post

Share on other sites

megahertz

Posted April 20, 2015 · Report post

lusca у меня хорошо кэширует такие ресурсы как megogo и аналоги, youtube кэшировать так я ее и не заставил.

он у меня банально jpeg не кэшит, можешь свои pattern'ы показать?

Sign In

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Share this post

Link to post

Share on other sites

Join the conversation