Welcome to Workbench’s documentation!¶

A medium-data framework for security research and development teams.¶

Workbench focuses on simplicity, transparency, and easy on-site customization. As an open source python project it provides light-weight task management, execution and pipelining for a loosely-coupled set of python classes.

Detailed Project Description¶

The workbench project takes the workbench metaphore seriously. It’s a platform that allows you to do work; it provides a flat work surface that supports your ability to combine tools (python modules) together. In general a workbench never constrains you (oh no! you can’t use those 3 tools together!) on the flip side it doesn’t hold your hand either. Using the workbench software is a bit like using a Lego set, you can put the pieces together however you want AND adding your own pieces is super easy!.

Workbench Client:¶

$ pip install zerorpc; echo 'Done!'

Workbench Server:¶

The indexers ‘Neo4j’ and ‘ElasticSearch’ are optional. We strongly suggest you install both of them but we also appreciate that there are cases where that’s not possible or feasible.

Install Indexers¶

Pull the repository¶

git clone https://github.com/supercowpowers/workbench.git

Warning!: The repository contains malcious data samples, be careful, exclude the workbench directory from AV, etc...

Install Python Modules¶

Note: Workbench is continuously tested with python 2.7. We’re currently working on Python 3 support (Issue 92).

• cd workbench
• pip install -r requirements.txt
• Go have a large cup of coffee...

Dependency Installation Errors¶

Python Modules

Note: If you get a bunch of clang errors about unknown arguments or ‘cannot link a simple C program’ add the following FLAGs:

```
$ export CFLAGS=-Qunused-arguments
$ export CPPFLAGS=-Qunused-arguments
```

Errors when running Tests

If when running the worker tests you get some errors like ‘MagicError: regexec error 17, (illegal byte sequence)’ it’s an issue with libmagic 5.17, revert to libmagic 5.16. Using brew on Mac:

$ cd /usr/local
$ brew versions libmagic # Copy the line for version 5.16, then paste (for me it looked like the following line)
$ git checkout bfb6589 Library/Formula/libmagic.rb
$ brew uninstall libmagic
$ brew install libmagic

Deprecated Stuff¶

Scapy Install

• brew tap Homebrew/python
• brew install scapy
• brew install pypcap
• If you get error about pyrex.distutils:
  • pip install pyrex (or if this doesn’t work do easy_install pyrex)
  • and then retry the ‘brew install pypcap’
• Still not working try pyrex from scatch pyrex

(2-5-14): For scapy python binding you have to manually install the latest release from secdev.org and follow the instructions (like first 5 lines)

$ wget http://www.secdev.org/projects/scapy/files/scapy-latest.zip
$ unzip scapy-latest.zip
$ cd scapy-2.*
$ sudo python setup.py install

Deprecated Instructions for Ubuntu 12.04¶

Server (localhost or server machine)¶

$ cd workbench
$ ./workbench

Example Clients (use -s for remote server)¶

There are about a dozen example clients showing how to use workbench on pcaps, PEfiles, pdfs, and log files. We even have a simple nodes.js client (looking for node devs to pop some pull requests :).

$ cd workbench/clients
$ python simple_workbench_client.py [-s tcp://mega.server.com]

Configuration File Information¶

When you first run workbench it copies default.ini to config.ini within the workbench/server directory, you can make local changes to this file without worrying about it getting overwritten on the next ‘git pull’. Also you can store API keys in it because it never gets pushed back to the repository.

# Example/default configuration for the workbench server
[workbench]

# Server URI (server machine ip or name)
# Example: mybigserver or 12.34.56.789
server_uri = localhost

# DataStore URI (datastore machine ip or name)
# Example: mybigserver or 12.34.56.789
datastore_uri = localhost

# Neo4j URI (Neo4j Graph DB machine ip or name)
# Example: mybigserver or 12.34.56.789
neo4j_uri = localhost

# ElasticSearch URI (ELS machine ip or name)
# Example: mybigserver or 12.34.56.789
els_uri = localhost

# DataStore Database
# Example: customer123, ml_talk, pdf_deep
database = workbench

# Storage Limits (in MegaBytes, 0 for no limit)
worker_cap = 10
samples_cap = 200

# VT API Key
# Example: 93748163412341234v123947
vt_apikey = 123

Workbench Examples¶

Please note that all of these notebooks are ‘clients’ hitting the workbench server. Making your own client is super easy! See Making a Client

• PCAP to Graph (A short teaser)
• Workbench Demo
• Adding a new Worker (super hawt)
• PCAP to Dataframe
• PCAP DriveBy Analysis
• Using Neo4j for PE File Sim Graph
• Generator Pipelines Notebook
• WIP Notebooks
  • Network Stream Analysis Notebook
  • PE File Static Analysis Notebook

import zerorpc
c = zerorpc.Client()
c.connect("tcp://127.0.0.1:4242")
with open('evil.pcap','rb') as f:
    md5 = c.store_sample('evil.pcap', f.read())
print c.work_request('pcap_meta', md5)

Running Tests¶

Unit testing and sub-pipeline tests

$ cd workbench/server/workers
$ ./runtests

Full pipeline tests (clients exercise a larger set of components)

$ cd workbench/clients
$ ./runtests

Test Coverage¶

If you want to run the test code coverage properly you’ll need to create a ~/.noserc file with these options:

[nosetests]
with-coverage=1
cover-erase=1
cover-inclusive=1
cover-min-percentage=90
cover-package=.

Benign Error¶

We have no idea why occasionaly you see this pop up in the server output. To our knowledge it literally has no impact on any functionality or robustness. If you know anything about this please help us out by opening an issue and pull request. :)

ERROR:zerorpc.channel:zerorpc.ChannelMultiplexer, unable to route event:
_zpc_more {'response_to': '67d7df3f-1f3e-45f4-b2e6-352260fa1507', 'zmqid':
['\x00\x82*\x01\xea'], 'message_id': '67d7df42-1f3e-45f4-b2e6-352260fa1507',
'v': 3} [...]

"VirusTotal Query Error, no valid response... past per min quota?"

Git Development Model¶

We’re going to use the ‘GitHub Flow’ model.

• To work on something new, create a descriptively named branch off of master (ie:new-oauth2-scopes)
• Commit to that branch locally and regularly push your work to the same named branch on the server
• When you need feedback or help, or you think the branch is ready for merging, open a pull request
• After someone else has reviewed and signed off on the feature, you can merge it into master

Git Example¶

$ git checkout -b my-awesome
$ git push -u origin my-awesome
$ <code for a bit>; git push
$ <code for a bit>; git push
$ Go to github and hit 'New pull request'