OpenStack

Is open source software for building private and public clouds.

Projects

Architecture

CI Environment

• Tests run on all proposed patches
  • Static Analysis
  • Unittests
  • Integration Tests
• Code merges are gated on tests
  • Rerun all tests
  • Merge only if tests pass again

Project Gating

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Tests run continuously
  • Large result dataset
  • Significant number of test artifacts

Test Results

• Binary Result States
  • Success
  • Failure
• Large amount of log data collected
  • Integration tests collect 366MB (uncompressed)
  • Can be used to determine more than two states

Log Archive

• Logs on disk
• Fronted by Apache + mod_autoindex
• Not indexed otherwise
• A lot of data, little information

Failures Happen

• Often not the fault of newly submitted code
• The 1% failure
  • Race Conditions
  • Underlying Hardware
• Dealt with via 'recheck' review comments

Recheck List

• Manually track bug:failure relationships
• High human overhead
• Incomplete data
• Prone to errors

Recheck List

Need Something Different

• Logs need to be accessible
  • Friendly UI
  • REST API
  • Query Language
• Data Pipeline
  • Flatten
  • Transform
  • Filter
  • Annotate
  • Realtime

Need Something Different

• Many Options
  • Fluentd
  • Hadoop
  • Splunk
  • Graylog
  • Reimann
  • Jenkins
  • Logstash
  • and more

Logstash, Kibana, ElasticSearch

• Open Source (Apache License v2)
• Simple UI
• REST API
• Lucene query language

Logstash, Kibana, ElasticSearch

• Flexible data processing pipeline
• Flatten disparate events into common schema
• Combine related events
• Filter noise
• Annotate events with additional information
• Near realtime (small delay)

Complications

• Collecting logs from untrusted Jenkins slaves
• Redis
• Impossible to index all logs

Design

• Jenkins ZeroMQ Event Publisher
• Gearman, process logs in batches one job per file
• Filter extensively

Design

What it gives us

• 1,293,256,221 log events indexed
• 72,621 jobs with indexed logs

ElasticRecheck

• Using bug fingerprints discover occurences of bugs
• Automates previous manual bug:failure relationship discovery
• Very accurate

ElasticRecheck

ElasticRecheck

ElasticRecheck

• Highly beneficial to Havana's release process
• Identified many specific race conditions
• Allowed for prioritization of work and confirmation that bugs were fixed

CRM114

• Difficult to find fingerprints (causes) for failures
• Better to have computers do that too
• Run log events through a spam filter
  • Classify events as most like Success or Failure
  • The CRM114 Discriminator

CRM114

• Orthogonal Sparse Bigram Classifier
• Variation on Markovian Classifier
  • Constructs unique word pairs with a five word window
  • Lighweight and Fast
• Probability phrase is in class determined using Bayesian Chain Rule

CRM114

• Train on Everything
• Jenkins preclassifies results for us, can learn against the proper set
• Logs for all jobs are similar, looking for items that do not belong
• Probabilistic Diffing

Results

• error_pr = log₁₀(Prob in Class) - log₁₀(Prob not in Class)
  • Negated on logs from failed jobs
• Positive error_pr: Success, > 10: strong correlation
• Negative error_pr: Failure, < -10: strong correlation

Results

• Query ElasticSearch for 'error_pr:[-1000.0 TO -10.0]'
• Capable of classifying errors that result in failure
  • 2013-12-27 23:37:02.388 7620 ERROR neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [req-9ca21083-ae6a-4978-9e07-fd188e1fa1fe bfcaa008905b41c4aaed19d54659f5b1 a7ba0994a0884e8485f6087108652f61] No DHCP agents are associated with network '65364310-6c0d-4d00-a7f4-602ae16f0cba'. Unable to send notification for 'subnet_create_end' with payload: {'subnet': {'name': u'tempest.scenario.manager-tempest-180461968-subnet', 'enable_dhcp': True, 'network_id': u'65364310-6c0d-4d00-a7f4-602ae16f0cba', 'tenant_id': u'52ed4d85ff514e7e9f598314ee69055b', 'dns_nameservers': [], 'allocation_pools': [{'start': '10.100.0.2', 'end': '10.100.0.14'}], 'host_routes': [], 'ip_version': 4, 'gateway_ip': '10.100.0.1', 'cidr': u'10.100.0.0/28', 'id': 'f7aa9538-a53a-4eef-abe2-ff3736e98a0f'}}
  • error_pr: -12.0434

Simple

# Apache License Version 2
window
isolate (:prefix:) /:*:_arg2:/
isolate (:target:) /:*:_arg3:/
learn [:_nl:] <osb unique microgroom> (:*:prefix:/SUCCESS.css)
learn [:_nl:] <osb unique microgroom> (:*:prefix:/FAILURE.css)
{
  window <bychar> /\n/ /\n/
  {
    isolate (:stats:)
    isolate (:result:)
    isolate (:prob:)
    isolate (:pr:)
    {
      {
        match (:timestamp:) /^[-.0-9 |:]+/
        alter (:timestamp:) //
      }
      learn <osb unique microgroom> (:*:prefix:/:*:target:.css)
      classify <osb unique microgroom> \
      (:*:prefix:/SUCCESS.css :*:prefix:/FAILURE.css) (:stats:)
      {
        match [:stats:] <nomultiline> \
        /^Best match to .*\/([A-Za-z]+).css\) prob: ([-.0-9]+)  pR: ([-.0-9]+)/ \
        ( :: :result: :prob: :pr: )
        {
          {
            match [:result:] /FAILURE/
            alter (:result:) /-/
          } alius {
            alter (:result:) //
          }
        }
        output /:*:result::*:pr:\n/
      }
    }
  }
  liaf
}
  

The Future

• Classifying log events is still very new
• Different training methods or classifiers may have better results
• Skynet

Thanks!

• Questions?
• Links:
  • Log Processor Pipeline
  • Logstash Configs
  • ElasticRecheck
  • CRM 114

These slides available at: http://docs.openstack.org/infra/publications/