{"id":1337,"date":"2023-04-10T12:16:31","date_gmt":"2023-04-10T10:16:31","guid":{"rendered":"https:\/\/dtstc.ugr.es\/neus-cslab\/?page_id=1337"},"modified":"2025-07-28T17:42:22","modified_gmt":"2025-07-28T15:42:22","slug":"ds-biblio","status":"publish","type":"page","link":"https:\/\/dtstc.ugr.es\/neus-cslab\/recursos\/ds-biblio\/","title":{"rendered":"Biblio-US17 – Dataset HTTP"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\n\t\t\t\tProgreso\t\t\t<\/span>\n\t\t\n\t\t
\n\t\t\t
\n\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t100%<\/span>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

BIBLIO-US17: Dataset HTTP<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t<\/i>\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tTipo contribuci\u00f3n\/resultado\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/h3>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\tDataset p\u00fablico\t\t\t\t\t<\/p>\n\t\t\t\t\n\t\t\t<\/div>\n\t\t\t\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t<\/i>\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tDescripci\u00f3n\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/h3>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\tDataset de peticiones HTTP reales etiquetadas para entrenamiento y validaci\u00f3n de AIDS y WAF
Incluye 47 millones de peticiones normales \/ ataques \/ err\u00f3neas
doi: 10.12795\/11441\/148254<\/a>\t\t\t\t\t<\/p>\n\t\t\t\t\n\t\t\t<\/div>\n\t\t\t\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t
\"\"<\/figure>

Este resultado ha sido parcialmente financiado por MCIN\/ AEI\/10.13039\/501100011033\/<\/p><\/div><\/div>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t
\"\"<\/figure>

Este resultado ha sido parcialmente financiado por FEDER\/ Junta de Andaluc\u00eda<\/p><\/div><\/div>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Presentaci\u00f3n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

El desarrollo y evaluaci\u00f3n de sistemas de detecci\u00f3n de intrusos basados en anomal\u00edas (AIDS) requiere de conjuntos de datos de entrenamiento<\/b> adecuados. Este nuevo conjunto de datos disponible para la comunidad cient\u00edfica\u00a0 tiene como objetivo posibilitar su uso en el contexto de los sistemas web.<\/p>\n

Ha sido recopilado a partir de las trazas del servidor web de la biblioteca de la Universidad de Sevilla<\/b> (Espa\u00f1a) y sus principales caracter\u00edsticas son:<\/p>\n

– Real<\/b>: ha sido adquirido a partir de la operaci\u00f3n real del servidor e incluye todas las peticiones recibidas.<\/p>\n

– Gran volumen de datos y lapso de tiempo<\/b>: incluye m\u00e1s de 47 millones de solicitudes HTTP recibidas por el servidor web p\u00fablico durante 6 meses.<\/p>\n

– Etiquetado<\/b> de peticiones: hemos seguido un proceso semiautom\u00e1tico para etiquetar cada registro en el conjunto de datos, tomando como punto de partida la clasificaci\u00f3n por varios SIDS.\nSe proporcionan as\u00ed peticiones leg\u00edtimas junto con ataques reales (es decir, en el mismo contexto del servidor web).<\/b><\/b><\/p>\n \n

El conjunto de datos obtenido puede utilizarse como ground-truth<\/em><\/b> para el entrenamiento y evaluaci\u00f3n<\/b> de sistemas de detecci\u00f3n de intrusiones a los servidores web. Este dataset presenta un alto potencial, ya que, adem\u00e1s de las caracter\u00edsticas mencionadas previamente, corresponde a un servicio altamente din\u00e1mico (amplio uso de queries<\/em> desplegado con par\u00e1metros) con tecnolog\u00eda actual.<\/p>\n

Para evitar problemas de privacidad, hemos restringido la informaci\u00f3n en el conjunto de datos al M\u00e9todo, recurso (URI) y c\u00f3digo de respuesta para cada solicitud, suprimiendo marcas temporales (pero manteniendo el orden secuencial), direcciones IP y reemplazando elementos sensibles en el URI.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Procedimiento<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAdquisici\u00f3n del dataset<\/strong><\/span>\n

Se ha adquirido todo el tr\u00e1fico a los servicios HTTP y HTTPS durante 6 meses a partir de los archivos de traza del servidor Apache. El tr\u00e1fico capturado se organiza en archivos diarios por servicio etiquetados con el n\u00famero de mes y d\u00eda. Se han agrupado estos archivos por d\u00eda y se ha a\u00f1adido un identificador \u00fanico a cada l\u00ednea. Los datos m\u00e1s relevantes de la adquisici\u00f3n son:<\/p>\n\n\n\n\n\n\n\n\n
Bloque<\/b><\/th>\nVol (GB)<\/b><\/th>\n# arch <\/b><\/th>\n# l\u00edneas<\/b><\/th>\n#Uris <\/b><\/th>\n<\/tr>\n<\/thead>\n
access_log<\/b><\/b><\/td>\n4.15<\/td>\n198<\/td>\n34 573 623<\/td>\n34 074 832<\/td>\n<\/tr>\n
ssl_access_log<\/b><\/td>\n1.17<\/td>\n172<\/td>\n13 328 700<\/td>\n13 328 164<\/td>\n<\/tr>\n
apachelog (ambos)<\/b><\/td>\n5.99<\/td>\n198<\/td>\n47 902 323<\/td>\n47 402 996<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n\n \n\n\n\n\n
\"\"<\/td>\n<\/td>\n\"\"<\/td>\n<\/tr>\n
Evoluci\u00f3n temporal del n\u00famero de peticiones capturadas por servicio<\/td>\n<\/td>\nDistribuci\u00f3n de peticiones por c\u00f3digo de respuesta<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n\n \n\nLimpieza y etiquetado<\/b><\/span>\n

Se han categorizado los registros (Normal \/ TP \/ FP \/ etc.) siguiendo el m\u00e9todo publicado en:<\/p>\n\n

<\/a><\/form>

D\u00edaz-Verdejo, Jes\u00fas E.; Estepa, Antonio; Estepa, Rafael; Madinabeitia, German; Mu\u00f1oz-Calle, Fco Javier<\/p>

A methodology for conducting efficient sanitization of HTTP training datasets<\/a> Art\u00edculo de revista<\/span> <\/p>

En: <\/span>Future Generation Computer Systems, <\/span>vol. 109, <\/span>pp. 67\u201382, <\/span>2020<\/span>, ISSN: 0167739X<\/span>.<\/p>

Resumen<\/a><\/span> | Enlaces<\/a><\/span> | BibTeX<\/a><\/span><\/p>

@article{Diaz-Verdejo2020,
\r\ntitle = {A methodology for conducting efficient sanitization of HTTP training datasets},
\r\nauthor = {Jes\u00fas E. D\u00edaz-Verdejo and Antonio Estepa and Rafael Estepa and German Madinabeitia and Fco Javier Mu\u00f1oz-Calle},
\r\nurl = {https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0167739X19322629},
\r\ndoi = {10.1016\/j.future.2020.03.033},
\r\nissn = {0167739X},
\r\nyear  = {2020},
\r\ndate = {2020-08-01},
\r\nurldate = {2020-08-01},
\r\njournal = {Future Generation Computer Systems},
\r\nvolume = {109},
\r\npages = {67--82},
\r\npublisher = {Elsevier B.V.},
\r\nabstract = {The performance of anomaly-based intrusion detection systems depends on the quality of the datasets used to form normal activity profiles. Suitable datasets should include high volumes of real-life data free from attack instances. On account of this requirement, obtaining quality datasets from collected data requires a process of data sanitization that may be prohibitive if done manually, or uncertain if fully automated. In this work, we propose a sanitization approach for obtaining datasets from HTTP traces suited for training, testing, or validating anomaly-based attack detectors. Our methodology has two sequential phases. In the first phase, we clean known attacks from data using a pattern-based approach that relies on tools that detect URI-based known attacks. In the second phase, we complement the result of the first phase by conducting assisted manual labeling systematically and efficiently, setting the focus of expert examination not on the raw data (which would be millions of URIs), but on the set of words that compose the URIs. This dramatically downsizes the volume of data that requires expert discernment, making manual sanitization of large datasets feasible. We have applied our method to sanitize a trace that includes 45 million requests received by the library web server of the University of Seville. We were able to generate clean datasets in less than 84 h with only 33 h of manual supervision. We have also applied our method to some public benchmark datasets, confirming that attacks unnoticed by signature-based detectors can be discovered in a reduced time span.},
\r\nkeywords = {},
\r\npubstate = {published},
\r\ntppubtype = {article}
\r\n}
\r\n<\/pre><\/div>
Cerrar<\/a><\/p><\/div>
The performance of anomaly-based intrusion detection systems depends on the quality of the datasets used to form normal activity profiles. Suitable datasets should include high volumes of real-life data free from attack instances. On account of this requirement, obtaining quality datasets from collected data requires a process of data sanitization that may be prohibitive if done manually, or uncertain if fully automated. In this work, we propose a sanitization approach for obtaining datasets from HTTP traces suited for training, testing, or validating anomaly-based attack detectors. Our methodology has two sequential phases. In the first phase, we clean known attacks from data using a pattern-based approach that relies on tools that detect URI-based known attacks. In the second phase, we complement the result of the first phase by conducting assisted manual labeling systematically and efficiently, setting the focus of expert examination not on the raw data (which would be millions of URIs), but on the set of words that compose the URIs. This dramatically downsizes the volume of data that requires expert discernment, making manual sanitization of large datasets feasible. We have applied our method to sanitize a trace that includes 45 million requests received by the library web server of the University of Seville. We were able to generate clean datasets in less than 84 h with only 33 h of manual supervision. We have also applied our method to some public benchmark datasets, confirming that attacks unnoticed by signature-based detectors can be discovered in a reduced time span.<\/div>
Cerrar<\/a><\/p><\/div>