Difference between revisions of "Gr-baz"

Revision as of 22:46, 5 April 2012

General

gr-baz is a GNU Radio project that adds new functionality (blocks, GRC definitions, apps, etc). It uses the standard GNU Radio block source tree layout and build process (i.e. ./configure, make and sudo make install).

Part of the new functionality makes it easier to analyse signals. For example, performing cyclostationary analyis (see Variable Delay block below) and Fast Auto-correlation. You can then do things like:

• Blind signal analysis of satellite transmissions
  • I presented this, along with my Mode S RADAR 3D airspace visualisation, in a talk entitled "Hacking the wireless world with Software Defined Radio" at Ruxcon 2011: http://www.youtube.com/watch?v=Vn-dpUegUDQ
• OFDM/DRM analysis
  • I put together a video on analysing signals on HF with these blocks in GNU Radio: http://www.youtube.com/watch?v=ZuRcaxpbYCw

NOTE: A few components will not work (e.g. BorIP client) without applying certain patches to your GNU Radio source. Please see below.

The module name is baz, so in Python one would write:

baz.<block>(args)

To import a file (e.g. borip):

from baz import <file>

If you make any improvements on this code, please get in touch!

Source code

You can get the master copy here with SVN, or browse with HTTP:

• http://svn.spench.net/main/gr-baz/

The code is also available on github if you prefer git (and perhaps want to fork it there):

• https://github.com/balint256/gr-baz

NOTE: You must have libusb-1.0 header+lib available! This is often easily installed using your favourite package manager. (I haven't yet had the time to fix the Automake configure script to check for it.)

Components

patch

Before using various blocks in gr-baz, you must apply at least some of the patches contained in this directory to your GNU Radio source tree. For example: you must update gr_udp_source if you wish to use it as a BorIP client.

lib (C++)

rtl_source_c

Updates

For updates, follow me @spenchdotnet.

For feedback, please get in touch.

Change list (SVN revision/git commit):

• r668/d03c816: Completely re-written, new GRC block

General

You can see a demo video of this block running from GRC here: http://youtu.be/FUQd9HOVTk8

If you want to use this on Windows, get the USRP Interfaces plugin and watch this one for installation instructions and a demo: http://youtu.be/Z0hEquzLsWU

This is a straight-forward block for receiving samples directly from a USB DVB-T stick that uses the:

• Realtek RTL2832 demodulator, and either of the following tuners:
  • Elonics E4000
  • Fitipower FC0013

(Support for FC0012 coming soon!)

The demodulator itself samples at 8-bit I/Q, and then this block re-processes this to output complex values (gr-complex).

It also supports:

• LNA gain control (the gain range depends on the tuner)
• for the E4000: optional automatic tuner mode control (enabled via auto_tuner_mode. This is activated when changing the gain and will calculate whether to use the nominal/sensitive/linear tuner mode).

Specs

The ezcap EzTV668 with E4000 tuner

Supported devices: USB DVB-T adapters (non-exhaustive list, see here for how to install device driver on Windows):

• The following are the names of some common devices, but others not listed here will work if their USB ID is the same:
  • ezcap
  • Hama nano
  • Terratec Noxon 1 & 2
  • Dexatek 1 (also Logilink/MSI)
  • Dexatek 2 (also ZAAPA HD)
  • Dexatek 3
  • Peak
  • Ardata MyVision/Gigabyte GT-U7300
  • MyGica/G-Tek
  • Lifeview
  • PROlectrix
  • Terratec Cinergy T 1
  • DIKOM HD
  • Twintech
  • Genius TVGo 2
  • SVEON

• Tuners are probed automatically now!

• You can override the tuner too, if you wish:
  • Linux: select the appropriate tuner from the list in the GRC block properties
  • Windows: add tuner=e4000/fc0012/fc0013/fc2580 to the device hint

• If your device is NOT recognised (missing from device table):
  • Linux: type in the VID/PID into the GRC block
  • Windows: add vid=0x#### pid=0x#### to device hint
  • Code: use the simple API to set these 'USB device override values'
  • Source: add one line to the device list in 'rtl2832.cc'

You can contact me with information (name/VID/PID/tuner type) regarding other devices that are not recognised automatically.

There is a helpful subreddit with a more complete list.

• Cross-platform source code: librtl2832++ is the C++ interface that abstracts the functionality of the hardware (demod & tuner). It compiles:
  • as a DLL on Windows
  • directly into the gr-baz baz module for GNU Radio

  You can freely take the rtl2832-* files from the lib directory out and re-use them in any other project where you need to control an RTL2832-based device!

Issues:

• Disabled E4000 DC offset loop to avoid period 'jumps' in time-domain signal: I have disabled the DC offset loop in the E4000 tuner as I was seeing additional crud on the right of the centre frequency. In the time-domain, you can see it as a period jump in the incoming signal. I'll add an option to re-enable it so you can experiment (or you can modify the E4000 init function and un-comment the function call).

• I2C errors when tuning rapidly: personally I haven't had this problem, except when using a GRC flowgraph where the down-stream sample rates are mismatched. However, other people have reported this in normal use. Therefore this issue is as yet unresolved, as perhaps because the device wasn't designed to be rapidly re-tuned. One option is rate-limiting tune requests and/or investigating async control transfers in libusb.

• I2C failure when tuning certain frequencies on E4000: For example, on both Windows and Linux if you tune to 90 MHz, then 89 MHz then back to 90 Mhz, you will see:

_I2CWriteByte: the control request was not supported by the device [-9] @ int PLL(rtl2832::tuner*, int, int):1749 "I2CReadByte(pTuner, 200, 5, writearray[0])"

• If you inspect the referenced E4000 PLL function, you will see an if blocks that set various registers depending on the requested tune frequency - this may hold the key. Otherwise, any other theories are welcome!

Notes

• Internally the source block is multithreaded to ensure smooth buffering of samples from the device, and delivery to GNU Radio's runtime.

• The source code itself is based on my USRP Interfaces plugin for Winrad/HDSDR/WRplus, which is in turn based on Osmocom's rtl-sdr. Please see the Osmocom page for more information on where to get an adapter, the original source code, etc.

• If the USB device is disconnected at runtime, the block detects this and signals EOF to the runtime.

• I have modified the operation of the E4000 tuner to yield (in my opinion) better receiver performance. Specifically I have fixed it to manual gain control and also disabled the DC offset loop as on my device it was the cause of annoying interference just to the right of 0 Hz (the LO).

• This block requires libusb-1.0 to compile. Please have the development files installed.

• The block outputs four short strings to notify you of streaming problems:
  • rO: Overflow inside libusb
  • rB: Overrun in streaming buffer (buffer consumption is too slow - have you set your resampling rates correctly?)
  • rT: The device did not yet submit new samples by the time it should have (i.e. the calculated libusb read interval + a little extra = 'Wait delay'), but the consumer is already asking for more samples - so we immediately give the consumer the samples from the buffer to avoid a pause in processing. Too many of these and...
  • rU: The internal buffer has underrun after too many consumer requests and not enough data coming in from the device via libusb. The block will now re-buffer to the 'Buffer level' (per cent) before sending any more samples to the consumer.

Permissions

To run with non-elevated privileges, choose <group name, e.g. 'dvb'>, and as root:

/usr/sbin/groupadd [group name]
/usr/sbin/usermod -G [group name] -a [user name]

Create the file:

/etc/udev/rules.d/10-[pick a name].rules

and add the following to it:

# rule to grant read/write access on USRP to group named usrp.
# to use, install this file in /etc/udev/rules.d as 10-usrp.rules
ACTION=="add", BUS=="usb", SYSFS{idVendor}=="[your VID]", SYSFS{idProduct}=="[your PID]", GROUP:="[group name]", MODE:="0660"

Then:

invoke-rc.d udev reload (or restart if you wish)

Have a look to see if the group for your device has changed (and mode should be crw-rw----):

ls -lR /dev/bus/usb

Then restart (unfortunately).

Issues

If, when you run the flowgraph, libgnuradio-baz.so.0 cannot be found, run:

sudo ldconfig

Interface

GR_SWIG_BLOCK_MAGIC(baz,rtl_source_c);

baz_rtl_source_c_sptr baz_make_rtl_source_c (bool defer_creation = false);

class baz_rtl_source_c : public gr_sync_block
{
private:
  baz_rtl_source_c(bool defer_creation = false);
public:
	void set_defaults();
	bool create(bool reset_defaults = false);
	void destroy();
public:
	void set_vid(/*uint16_t*/int vid);
	void set_pid(/*uint16_t*/int pid);
	void set_default_timeout(int timeout);	// 0: use default, -1: poll only
	void set_fir_coefficients(const std::vector</*uint8_t*/int>& coeffs);
	void set_crystal_frequency(uint32_t freq);
	void set_tuner_name(const char* name);
public:
	size_t recv_samples_per_packet() const;
	uint64_t samples_received() const;
	uint32_t overflows() const;
	bool running() const;
	uint32_t buffer_size() const;
	uint32_t buffer_times() const;
	bool buffering() const;
	uint32_t read_packet_count() const;
	uint32_t buffer_overflow_count() const;
	uint32_t buffer_underrun_count() const;
public:
	void set_verbose(bool on = true);
	void set_read_length(/*uint32_t*/int length);
	void set_buffer_multiplier(/*uint32_t*/int mul);
	void set_use_buffer(bool use = true);
	void set_buffer_level(float level);
public:
	bool relative_gain() const;
	bool verbose() const;
	uint32_t read_length() const;
	uint32_t buffer_multiplier() const;
	bool use_buffer() const;
	float buffer_level() const;
public:
	bool set_sample_rate(double sample_rate);
	bool set_frequency(double freq);
	bool set_gain(double gain);
	bool set_bandwidth(double bandwidth);
	bool set_gain_mode(int mode);
	bool set_gain_mode(const char* mode);
	void set_relative_gain(bool on = true);
	int set_auto_gain_mode(bool on = true);
public:
	double sample_rate() const;
	RTL2832_NAMESPACE::range_t sample_rate_range() const;
	double frequency() const;
	double gain() const;
	double bandwidth() const;
	int gain_mode() const;
	std::string gain_mode_string() const;
	bool auto_gain_mode() const;
public:	// SWIG get: tuner ranges/values
	RTL2832_NAMESPACE::range_t gain_range() const;
	RTL2832_NAMESPACE::values_t gain_values() const;
	RTL2832_NAMESPACE::range_t frequency_range() const;
	RTL2832_NAMESPACE::range_t bandwidth_range() const;
	RTL2832_NAMESPACE::values_t bandwidth_values() const;
	RTL2832_NAMESPACE::num_name_map_t gain_modes() const;
	std::pair<bool,int> calc_appropriate_gain_mode()/* const*/;
};

Screenshots

Channel selection and FM demodulation with RTL2832 Source

RTL2832 Source feeding new OP25 Decoder block

Example of the GRC Source block in a flowgraph

The above flowgraph running

delay

Unlike the original gr_delay block that has a fixed delay, this allows for a variable delay that can be changed at runtime. The delay can be positive or negative. If any padding is required, it uses the value of the last sample.

This block is particularly useful for investigating periodic repetitions within signals (e.g. adjusting cyclostationary lag when performing blind analysis on an OFDM signal).

GR_SWIG_BLOCK_MAGIC(baz,delay)

baz_delay_sptr baz_make_delay (size_t itemsize, int delay);

class baz_delay : public gr_sync_block
{
 private:
  baz_delay (size_t itemsize, int delay);

 public:
  int  delay() const;
  void set_delay (int delay);
};

Using the Variable Delay block to adjust the delay of samples into Complex Conjugate/Multiply blocks in order to find the cyclostationarity of the input signal (i.e. perform cyclostationary analysis to determine the periodicity of the signal to find the symbol or baud rate of the raw modulated data - helpful for blind signal anaylsis).

pow_cc

Raise incoming signal to a power (exponent), and optionally divide resulting signal by 10^div_exp to ensure it doesn't go too high and result in NaNs/Infinity in subsequent floating-point calculations.

GR_SWIG_BLOCK_MAGIC(baz,pow_cc);

baz_pow_cc_sptr
baz_make_pow_cc (float exponent, float div_exp = 0.0);

class baz_pow_cc : public gr_sync_block
{
  baz_pow_cc (float exponent, float div_exp = 0.0);
public:
  void set_exponent(float exponent);
  void set_division_exponent(float div_exp);
  float exponent() const;
  float division_exponent() const;
};

Using the Power block to determine the order of a PSK signal (i.e. determine the n of a phase-shift keyed signal, e.g. BPSK, QPSK, 8PSK).

print_char

Print bytes to the console/stdout (or optionally a file). The hexadecimal values of the bytes are printed, rather than the raw character interpretation of the value.

This block accepts a byte-stream input (containing the data to print), and an optional float-stream (a variable length signal to be used as the trigger for printing). The threshold can be set to only print bytes when the values in the float-stream exceed the specified threshold. To prevent too much data being printed, set limit to the total number of characters to be printed. This is reset each time the float-stream drops below the trigger threshold (i.e. it can be used to print the first limit demodulated bytes when the receiver becomes unsquelched, e.g. VDL Mode 2).

GR_SWIG_BLOCK_MAGIC(baz,print_char);

baz_print_char_sptr baz_make_print_char (float threshold = 0.0, int limit = -1, const char* file = NULL);

class baz_print_char : public gr_sync_block
{
private:
  baz_print_char (float threshold, int limit, const char* file);
};

puncture_bb/depuncture_ff

Puncture a stream of bytes after Forward Error Correction to reduce the code rate from that of the mother code.

Also, depuncture a stream of floats (symbols) to restore the rate of the received code to that of the mother code prior to detecting the code for errors. Missing symbol slots are marked by erasures (the value 0 is inserted into the outgoing float symbol stream).

The matrix parameter can be set at design and runtime. Here are some examples taken from autofec (see below):

_puncture_matrices = [ # Format is ('name', matrix, rate ratio)
  ('1/2', [1,1], (1, 2)),
  ('2/3', [1,1,0,1], (2, 3)),
  ('3/4', [1,1,0,1,1,0], (3, 4)),
  ('5/6', [1,1,0,1,1,0,0,1,1,0], (5, 6)),
  ('7/8', [1,1,0,1,0,1,0,1,1,0,0,1,1,0], (7, 8)),
  ('2/3*', [1,1,1,0], (2, 3)),
  ('3/4*', [1,1,1,0,0,1], (3, 4)),
  ('5/6*', [1,1,1,0,0,1,1,0,0,1], (5, 6)),
  ('7/8*', [1,1,1,0,1,0,1,0,0,1,1,0,0,1], (7, 8))

GR_SWIG_BLOCK_MAGIC(baz,puncture_bb)

baz_puncture_bb_sptr baz_make_puncture_bb (const std::vector<int>& matrix);

class baz_puncture_bb : public gr_block
{
 private:
  baz_puncture_bb (const std::vector<int>& matrix);

 public:
  void set_matrix (const std::vector<int>& matrix);
};

GR_SWIG_BLOCK_MAGIC(baz,depuncture_ff)

baz_depuncture_ff_sptr baz_make_depuncture_ff (const std::vector<int>& matrix);

class baz_depuncture_ff : public gr_block
{
 private:
  baz_depuncture_ff (const std::vector<int>& matrix);

 public:
  void set_matrix (const std::vector<int>& matrix);
};

swap_ff

Swap pairs of incoming float samples (if disabled, acts in pass-through mode).

GR_SWIG_BLOCK_MAGIC(baz,swap_ff)

baz_swap_ff_sptr baz_make_swap_ff (bool bSwap);

class baz_swap_ff : public gr_sync_block
{
 private:
  baz_swap_ff (bool bSwap);

 public:
  void set_swap (bool bSwap);
};

test_counter_cc

Used to check a simulated complex stream. The I and Q values are not a real signal; they are values that should count upward together. If values are dropped, or go out of sync, this block will report problems to stderr.

A source of such a test stream is the ExtIO_USRP plugin. To enable 'test mode', add the following DWORD to the registry:

HKEY_CURRENT_USER\<Winrad/HDSDR/WRplus/etc>\ExtIO_USRP\Settings\m_bTestMode = 1

GR_SWIG_BLOCK_MAGIC(baz,test_counter_cc);

baz_test_counter_cc_sptr baz_make_test_counter_cc ();

class baz_test_counter_cc : public gr_sync_block
{
  baz_test_counter_cc ();
};

unpacked_to_packed_bb

This is the missing combination of operations of those blocks included in the GNU Radio source tree. This one here will take bits_per_chunk from the incoming byte stream and reassemble outgoing bytes to contain bits_into_output bits from the original stream. For example: in VDL Mode 2 the D8PSK demodulator outputs an unpacked byte stream (which is really the equivalent of a bit stream). This block can be used to reassemble the individual 8PSK symbols into individual bytes (i.e. 3 bits in a byte, so output value ranges from 0 to 7).

GR_SWIG_BLOCK_MAGIC(baz,unpacked_to_packed_bb);

baz_unpacked_to_packed_bb_sptr
baz_make_unpacked_to_packed_bb (unsigned int bits_per_chunk, unsigned int bits_into_output, /*gr_endianness_t*/int endianness = GR_MSB_FIRST);

class baz_unpacked_to_packed_bb : public gr_block
{
  baz_unpacked_to_packed_bb (unsigned int bits_per_chunk, unsigned int bits_into_output, /*gr_endianness_t*/int endianness);
};

agc_cc

Experimental and unfinished.

python

borip

NOTE: You must apply the appropriate patch to enable BorIP support in gr_udp_source first!

Enables BorIP client support in GNU Radio. Either access this class directly in hand-written Python code, via the GRC block (see below), or enable seamless connection to a remote USRP exposed to your LAN with BorIP server.

BorIP has some neat features for using a USRP 1 over a network (more details in the BorIP specification):

• Data packets contain a header (optional, enabled by default) that enables the receiver to keep track of capture overruns on the server, and packets lost on the network.
• Transparent operation (see below)
• Automatically handle connection interruption without affecting the flowgraph's operation: if contact is lost with the BorIP server (e.g. network is unplugged, server power lost, etc), the UDP data stream would stop, and eventually the server would stop sending data and disconnect after the TCP control connection closes. The BorIP client can be configured to automatically reconnect to the server in the background and start streaming data again as if nothing had happened.

The UDP receiving code will process BorIP packets and output losses (bO) and incorrect payload sizes (b!) to stderr. uO is also output when the remote server experiences a capture overrun.

Operational modes

• Traditional UDP Source that handles BorIP data packets
  • NOTE: There is no TCP control channel established (unlike the proper BorIP Source block), so control commands cannot be sent to a server. This mode is suited for use with the UDP Relay option in the ExtIO_USRP+FUNcube Dongle plugin.

The original UDP Source block with added BorIP support (also features the Test Counter block)

• Proper BorIP Source that also establishes a control channel to communicate radio configuration to the server
  • This block operates more like a traditional USRP Source block in that one can configure parameters such as sample rate, frequency, gain and antenna:

To enable seamless support, add the following to gr-usrp/src/__init__.py:

import sys
if not sys.modules.has_key('baz.borip'):
    from baz import borip

This tells Python to import borip whenever the Python-bindings for the USRP are requested. BorIP automatically patches into usrp_source_c and will attempt to connect to a BorIP server if no local device is present. The default server address is specified in ~/.gnuradio/config.conf:

[borip]
server=<server address>

The other settings are:

reconnect_attempts=<# of reconnection attempts before signalling EOF to the flowgraph, -1 to try forever, default is 0>
reconnect_interval=<seconds between connection attempts, default 5 seconds>
keepalive_interval=<seconds between sending PING keepalive, default is 5 seconds>
verbose={True,False} Verbose mode prints commands sent between client and server
default_port=<TCP and UDP port to use, default is 28888>

If you use the BorIP Source GRC block, the generated code will NOT honour the reconnect_attempts specified in the above config file. Instead it will always take it from the parameter set for the GRC block instance.

auto_fec

Automatically try every combination of FEC parameters until the correct ones are found (determined by monitoring BER output by Viterbi decoder).

This uses the NASA Voyager (k=7) convolutional code.

Leave 'sample rate' parameter at 0 to have it interpret duration/periods as number of samples (this should work fine in all cases anyway). If you specify the sample rate, you can optionally engage the internal Throttle if playing back from a file.

eye

This draws upon the Data Scope originally by Max (KA1RBI) from OP25. The block allows for changing certain sampling/graphical parameters at runtime (see GRC block definition below).

Eye diagram showing 2-level signal

facsink

Please see Fast Auto-correlation.

Fast Auto-correlation block at bottom-right

grc

GRC XML block definitions for:

• rtl_source_c: RTL2832 Source
• agc: AGC (Baz) - this is unfinished
• auto_fec: Auto FEC
• borip_source: BorIP Source
• delay: Variable Delay
• puncture_bb/depuncture_ff: Puncture/Depuncture
• pow_cc: Power
• swap_ff: Swap
• test_counter_cc: Test Counter
• eye: Eye Diagram
• facsink: Fast AutoCorrelation Sink
• gr_mpsk_receiver_debug_cc: MPSK Receiver (Debug)
  • Requires appropriate patch to gr_mpsk_receiver_debug.cc

Also, this contains patches for GRC to enable 'any' block support. This enables you to use raw GNU Radio blocks by typing in the necessary Python function to create the block (without a dedicated GRC XML block definition). For example: I use it with baz.print_char

• baz_any_source: Any Block Source (for source blocks)
• baz_any_sink: Any Block Sink (for sink blocks)
• baz_any: Any Block (for blocks with I/O)

You can either apply the following two patches, OR apply one from the patch page:

• Block.py.diff
• Port.py.diff

Examples of GRC Any Blocks

apps

am_fft

The AM Scope has the same functionality as usrp_fft.py, however the AM (magnitude) signal is shown instead of the original complex one.

Also allows for changing receive antenna in the GUI.

Mode S transmission in AM

usrp_fac

Stand-alone app for using the Fast Auto-correlation Sink.

W-CDMA mobile signal (this is actually a GRC-generated GUI, but the stand-alone app look similar and can be configured with command-line arguments)