Ship access log to ElasticSearch

Posted by 4Aiur on 2012/08/13 in Python |

Ship access log to ElasticSearch

This article introduce how to use a custom python script to parse Apache access log and shipping it to ElasticSearch.
If you wan’t store the huge log to ElasticSearch, you should read Using Elasticsearch for logs, Using some popular OpenSource software, like Graylog2, Logstash, Apache Flume.

System basic setup

Installation and Configuration ElasticSearch

Deploying ElasticSearch on a Cluster(EC2)

Cluster status

Schema Mapping

Ship log script

I have put this code on the github source code.

Usage:

or

config file:

code:

Related links:

  • elasticsearch, ElasticSearch is an Open Source (Apache 2), Distributed, RESTful, Search Engine built on top of Apache Lucene.
  • elasticsearch-head, elasticsearch-head is a web front end for browsing and interacting with an Elastic Search cluster.
  • pyes, pyes is a connector to use elasticsearch from python.

Tags:

Copyright © 2010-2017 4Aiur All rights reserved.
This site is using the Desk Mess Mirrored theme, v2.5, from BuyNowShop.com.

.