Varability in the Content of Internet Forums
DOI:
https://doi.org/10.18559/54ecxh10Keywords:
Internet forums, Internet, InformatizationAbstract
In this article we present the results of a study conducted on a sample of Polish Web forums in order to investigate how these sites evolve over time. We analysed more than 27 900 Web pages from 16 sources at two hour intervals (4 256 data points) over 22 days of the experiment. Th e results can be the basis for improving Web crawler design, providing valuable insights into the nature of Web forums. It appears that the variability of Web forums content is signifi cantly diff erent from general-purpose Web sites, thus Web crawlers need to adjust their document extraction policies to deal with this kind of Web source.
Downloads
References
Adar, E., Teevan, J. i Dumais, S.T., 2009, Resonance on the Web: Web dynamics and Revisitation patterns, w: Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI '09, ACM, New York, NY, USA, s. 1381-1390.
View in Google Scholar
Adar, E., Teevan, J., Dumais, S.T. i Elsas, J.L., 2009, Th e Web Changes Everything: Understanding the Dynamics of Web Content, w: Proceedings of the Second ACM International Conference on Web Search and Data Mining, WSDM '09, ACM, New York, NY, USA, s. 282-291.
View in Google Scholar
Baeza-Yates, R. i Castillo, C., 2007, Crawling the Infi nite Web, J. Web Eng, vol. 6, no. 1, s. 49-72.
View in Google Scholar
Baeza-Yates, R., Castillo, C., Marin, M. i Rodriguez, A., 2005, Crawling a Country: Better Strategies than Breadth-first for Web Page Ordering, w: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, WWW '05, ACM, New York, NY, USA.
View in Google Scholar
Ben Saad, M. i Gaŋcarski, S., 2011, Archiving the Web Using Page Changes Patterns: a Case Study, w: Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, JCDL '11, ACM, New York, NY, USA.
View in Google Scholar
Buttler, D., Rocco, D. i Liu, L., 2004, Efficient Web Change Monitoring with Page Digest, w: Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters, WWW Alt. '04, ACM, New York, NY, USA, s. 476-477.
View in Google Scholar
Cai, R., Yang, J.M., Lai, W., Wang, Y. i Zhang, L., 2008, iRobot: An Intelligent Crawler for Web Forums, w: Proceedings of the 17th International Conference on World Wide Web, WWW'08, ACM, New York, NY, USA, s. 447-456.
View in Google Scholar
Cho, J. i Garcia-Molina, H., 2000, Th e Evolution of the Web and Implications for an Incremental Crawler, w: Proceedings of the 26th International Conference on Very Large Data Bases, VLDB '00, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, s. 200-209.
View in Google Scholar
Cho, J. i Garcia-Molina, H., 2003, Estimating Frequency of Change, ACM Trans. Internet Technol., vol. 3, no. 3, s. 256-290.
View in Google Scholar
Douglis, F. i Ball, T., 1996, Tracking and Viewing Changes on the Web, w: USENIX Technical Conference, AT&T Bell Laboratories.
View in Google Scholar
Douglis, F., Ball, T., Chen, Y.F. i Koutsofi os, E., 1998, The AT&T Internet Difference Engine: Tracking and Viewing Changes on the Web, World Wide Web, vol. 1, s. 27-44.
View in Google Scholar
Farn Chen, Y., Douglis, F., Huang, H. i phong Vo, K., 2000, TopBlend: An Efficient Implementationof HtmlDiff in Java, w: World Conference on the WWW and Internet, s. 88-94.
View in Google Scholar
Hirschberg, D.S., 1977, Algorithms for the Longest Common Subsequence Problem, J. ACM, vol. 24, no. 4, s. 664-675.
View in Google Scholar
Jacobson, G. i Vo, K.P., 1992, Heaviest Increasing/Common Subsequence Problems, w: A. Apostolico, M. Crochemore, Z. Galil i U. Manber (eds.), Combinatorial Pattern Matching, tom 644 z serii Lecture Notes in Computer Science, Springer Berlin / Heidelberg, s. 52-66.
View in Google Scholar
Jiang, J., Yu, N. i Lin, C.Y., 2012, FoCUS: Learning to Crawl Web Forums, w: Proceedings of the 21st International Conference Companion on World Wide Web, WWW '12 Companion, ACM, New York, NY, USA, s. 33-42.
View in Google Scholar
Kwon, S., Lee, S. i Kim, S., 2006, Effective Criteria for Web Page Changes, w: X. Zhou, J. Li, H. Shen, M. Kitsuregawa i Y. Zhang (eds.), Frontiers of WWW Research and Development - APWeb 2006, t. 3841 z serii Lecture Notes in Computer Science, Springer Berlin / Heidelberg.
View in Google Scholar
Law, M.T., Th ome, N., Gaŋcarski, S. i Cord, M., 2012, Structural and Visual Comparisons for Web Page Archiving, w: Proceedings of the 2012 ACM Symposium on Document Engineering, DocEng '12, ACM, New York, NY, USA, s. 117-120.
View in Google Scholar
Liu, M., Cai, R., Zhang, M. i Zhang, L., 2011, User Browsing Behavior-driven Web Crawling, w: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM '11, ACM, New York, NY, USA, s. 87-92.
View in Google Scholar
Rocco, D., Buttler, D. i Liu, L., 2003, Page Digest for Large-scale Web Services, w: E-Commerce, 2003. CEC 2003. IEEE International Conference on, s. 381-390.
View in Google Scholar
Saad, M.B. i Gaŋcarski, S., 2010, Using Visual Pages Analysis for Optimizing Web Archiving, w: Proceedings of the 2010 EDBT/ICDT Workshops, EDBT '10, ACM, New York, NY, USA, s. 43:1-43:7.
View in Google Scholar
Toyoda, M. i Kitsuregawa, M., 2006, What's Really New on the Web?: Identifying New Pages from a Series of Unstable Web Snapshots, w: Proceedings of the 15th International Conference on World Wide Web, WWW '06, ACM, New York, NY, USA, s. 233-241.
View in Google Scholar
Yang, J.M., Cai, R., Wang, C., Huang, H., Zhang, L. i Ma, W.Y., 2009, Incorporating Site-level Knowledge for Incremental Crawling of Web Forums: A List-wise Strategy, w: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, ACM, New York, NY, USA, s. 1375-1384.
View in Google Scholar
Yeh, P.J., Li, J.T. i Yuan, S.M., 2006, Tracking the Changes of Dynamic Web Pages in the Existence of URL Rewriting, w: Proceedings of the Fift h Australasian Conference on Data Mining and Analystics - Volume 61, AusDM '06, Australian Computer Society, Inc., Darlinghurst, Australia, Australia, s. 169-176.
View in Google Scholar
