Sensors reference

  • Collision detector
  • Depth photographic camera
  • GNSS sensor
  • IMU sensor
  • Lane invasion detector
  • LIDAR sensor
  • Obstacle detector
  • Radar sensor
  • RGB camera
  • RSS sensor
  • Semantic LIDAR sensor
  • Semantic segmentation photographic camera
  • DVS photographic camera
  • Optical Flow camera

Important

All the sensors use the UE coordinate arrangement (x-frontward, y-right, z-up), and render coordinates in local space. When using whatever visualization software, pay attention to its coordinate system. Many invert the Y-axis, so visualizing the sensor information straight may issue in mirrored outputs.

Collision detector

  • Blueprint: sensor.other.standoff
  • Output: carla.CollisionEvent per collision.

This sensor registers an issue each fourth dimension its parent actor collisions against something in the earth. Several collisions may be detected during a single simulation step. To ensure that collisions with whatsoever kind of object are detected, the server creates "fake" actors for elements such as buildings or bushes so the semantic tag can exist retrieved to identify information technology.

Collision detectors do not accept any configurable attribute.

Output attributes

Sensor data attribute Type Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the outset of the episode.
transform carla.Transform Location and rotation in earth coordinates of the sensor at the time of the measurement.
actor carla.Role player Actor that measured the standoff (sensor's parent).
other_actor carla.Thespian Actor against whom the parent collided.
normal_impulse carla.Vector3D Normal impulse upshot of the collision.

Depth photographic camera

  • Design: sensor.photographic camera.depth
  • Output: carla.Image per stride (unless sensor_tick says otherwise).

The camera provides a raw information of the scene codifying the distance of each pixel to the camera (also known as depth buffer or z-buffer) to create a depth map of the elements.

The prototype codifies depth value per pixel using 3 channels of the RGB color space, from less to more than significant bytes: R -> Thousand -> B. The actual distance in meters can be decoded with:

          normalized = (R + K * 256 + B * 256 * 256) / (256 * 256 * 256 - 1) in_meters = 1000 * normalized

The output carla.Prototype should then exist saved to deejay using a carla.colorConverter that will turn the altitude stored in RGB channels into a [0,1] bladder containing the distance so translate this to grayscale. There are two options in carla.colorConverter to become a depth view: Depth and Logaritmic depth. The precision is milimetric in both, just the logarithmic arroyo provides better results for closer objects.

          ... raw_image.save_to_disk("path/to/save/converted/paradigm",carla.Depth)

ImageDepth

Basic camera attributes

Blueprint aspect Type Default Description
image_size_x int 800 Epitome width in pixels.
image_size_y int 600 Paradigm tiptop in pixels.
fov float ninety.0 Horizontal field of view in degrees.
sensor_tick bladder 0.0 Simulation seconds between sensor captures (ticks).

Camera lens distortion attributes

Blueprint attribute Type Default Description
lens_circle_falloff bladder 5.0 Range: [0.0, ten.0]
lens_circle_multiplier float 0.0 Range: [0.0, ten.0]
lens_k float -1.0 Range: [-inf, inf]
lens_kcube bladder 0.0 Range: [-inf, inf]
lens_x_size float 0.08 Range: [0.0, 1.0]
lens_y_size float 0.08 Range: [0.0, 1.0]

Output attributes

Sensor information attribute Type Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the commencement of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the time of the measurement.
width int Image width in pixels.
height int Image pinnacle in pixels.
fov float Horizontal field of view in degrees.
raw_data bytes Array of BGRA 32-bit pixels.

GNSS sensor

  • Blueprint: sensor.other.gnss
  • Output: carla.GNSSMeasurement per step (unless sensor_tick says otherwise).

Reports current gnss position of its parent object. This is calculated past adding the metric position to an initial geo reference location defined within the OpenDRIVE map definition.

GNSS attributes

Design aspect Type Default Description
noise_alt_bias float 0.0 Hateful parameter in the noise model for altitude.
noise_alt_stddev bladder 0.0 Standard deviation parameter in the noise model for altitude.
noise_lat_bias bladder 0.0 Mean parameter in the noise model for latitude.
noise_lat_stddev float 0.0 Standard deviation parameter in the racket model for breadth.
noise_lon_bias float 0.0 Mean parameter in the noise model for longitude.
noise_lon_stddev bladder 0.0 Standard departure parameter in the noise model for longitude.
noise_seed int 0 Initializer for a pseudorandom number generator.
sensor_tick bladder 0.0 Simulation seconds between sensor captures (ticks).

Output attributes

Sensor information attribute Type Clarification
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the start of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the time of the measurement.
breadth double Latitude of the histrion.
longitude double Longitude of the role player.
altitude double Distance of the actor.

IMU sensor

  • Blueprint: sensor.other.imu
  • Output: carla.IMUMeasurement per step (unless sensor_tick says otherwise).

Provides measures that accelerometer, gyroscope and compass would retrieve for the parent object. The data is collected from the object'southward current country.

IMU attributes

Blueprint attribute Blazon Default Description
noise_accel_stddev_x float 0.0 Standard difference parameter in the noise model for dispatch (Ten centrality).
noise_accel_stddev_y float 0.0 Standard deviation parameter in the noise model for dispatch (Y centrality).
noise_accel_stddev_z float 0.0 Standard divergence parameter in the noise model for acceleration (Z axis).
noise_gyro_bias_x bladder 0.0 Mean parameter in the noise model for the gyroscope (X centrality).
noise_gyro_bias_y float 0.0 Mean parameter in the noise model for the gyroscope (Y axis).
noise_gyro_bias_z float 0.0 Mean parameter in the dissonance model for the gyroscope (Z axis).
noise_gyro_stddev_x float 0.0 Standard deviation parameter in the dissonance model for the gyroscope (X centrality).
noise_gyro_stddev_y float 0.0 Standard deviation parameter in the racket model for the gyroscope (Y centrality).
noise_gyro_stddev_z bladder 0.0 Standard deviation parameter in the noise model for the gyroscope (Z axis).
noise_seed int 0 Initializer for a pseudorandom number generator.
sensor_tick float 0.0 Simulation seconds between sensor captures (ticks).

Output attributes

Sensor data attribute Blazon Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the starting time of the episode.
transform carla.Transform Location and rotation in globe coordinates of the sensor at the fourth dimension of the measurement.
accelerometer carla.Vector3D Measures linear acceleration in k/south^2.
gyroscope carla.Vector3D Measures angular velocity in rad/sec.
compass float Orientation in radians. North is (0.0, -1.0, 0.0) in UE.

Lane invasion detector

  • Pattern: sensor.other.lane_invasion
  • Output: carla.LaneInvasionEvent per crossing.

Registers an consequence each time its parent crosses a lane marker. The sensor uses road data provided by the OpenDRIVE description of the map to decide whether the parent vehicle is invading another lane by considering the space between wheels. Withal there are some things to be taken into consideration:

  • Discrepancies between the OpenDRIVE file and the map will create irregularities such as crossing lanes that are not visible in the map.
  • The output retrieves a list of crossed lane markings: the ciphering is done in OpenDRIVE and considering the whole infinite between the four wheels as a whole. Thus, there may exist more than 1 lane being crossed at the same time.

This sensor does non take any configurable aspect.

Important

This sensor works fully on the client-side.

Output attributes

Sensor information attribute Type Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the beginning of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the time of the measurement.
actor carla.Thespian Vehicle that invaded another lane (parent actor).
crossed_lane_markings listing(carla.LaneMarking) List of lane markings that have been crossed.

LIDAR sensor

  • Blueprint: sensor.lidar.ray_cast
  • Output: carla.LidarMeasurement per step (unless sensor_tick says otherwise).

This sensor simulates a rotating LIDAR implemented using ray-casting. The points are computed by adding a laser for each aqueduct distributed in the vertical FOV. The rotation is simulated computing the horizontal angle that the Lidar rotated in a frame. The betoken deject is calculated by doing a ray-bandage for each laser in every step. points_per_channel_each_step = points_per_second / (FPS * channels)

A LIDAR measurement contains a package with all the points generated during a ane/FPS interval. During this interval the physics are non updated so all the points in a measurement reflect the same "static picture" of the scene.

This output contains a cloud of simulation points and thus, it can be iterated to retrieve a list of their carla.Location:

          for location in lidar_measurement:     print(location)

The data of the LIDAR measurement is enconded 4D points. Beingness the first three, the space points in xyz coordinates and the last one intensity loss during the travel. This intensity is computed by the following formula.
LidarIntensityComputation

a — Attenuation coefficient. This may depend on the sensor's wavelenght, and the conditions of the temper. It can exist modified with the LIDAR attribute atmosphere_attenuation_rate. d — Altitude from the hitting point to the sensor.

For a ameliorate realism, points in the cloud tin can be dropped off. This is an piece of cake manner to simulate loss due to external perturbations. This tin done combining 2 unlike.

  • General drop-off — Proportion of points that are dropped off randomly. This is done before the tracing, meaning the points being dropped are not calculated, and therefore improves the performance. If dropoff_general_rate = 0.five, half of the points will be dropped.
  • Instensity-based drib-off — For each betoken detected, and extra drop-off is performed with a probability based in the computed intensity. This probability is adamant by 2 parameters. dropoff_zero_intensity is the probability of points with zero intensity to be dropped. dropoff_intensity_limit is a threshold intensity to a higher place which no points will be dropped. The probability of a point within the range to be dropped is a linear proportion based on these two parameters.

Additionally, the noise_stddev attribute makes for a dissonance model to simulate unexpected deviations that appear in real-life sensors. For positive values, each point is randomly perturbed along the vector of the laser ray. The outcome is a LIDAR sensor with perfect athwart positioning, but noisy distance measurement.

The rotation of the LIDAR can be tuned to encompass a specific angle on every simulation step (using a fixed time-stride). For instance, to rotate once per step (full circle output, as in the film below), the rotation frequency and the faux FPS should be equal.
1. Set the sensor'southward frequency sensors_bp['lidar'][0].set_attribute('rotation_frequency','10').
2. Run the simulation using python3 config.py --fps=x.

LidarPointCloud

Lidar attributes

Pattern attribute Type Default Description
channels int 32 Number of lasers.
range float 10.0 Maximum distance to measure/raycast in meters (centimeters for CARLA 0.9.vi or previous).
points_per_second int 56000 Points generated by all lasers per second.
rotation_frequency bladder 10.0 LIDAR rotation frequency.
upper_fov float 10.0 Angle in degrees of the highest light amplification by stimulated emission of radiation.
lower_fov float -30.0 Angle in degrees of the lowest laser.
horizontal_fov float 360.0 Horizontal field of view in degrees, 0 - 360.
atmosphere_attenuation_rate float 0.004 Coefficient that measures the LIDAR instensity loss per meter. Cheque the intensity computation above.
dropoff_general_rate bladder 0.45 General proportion of points that are randomy dropped.
dropoff_intensity_limit float 0.8 For the intensity based driblet-off, the threshold intensity value in a higher place which no points are dropped.
dropoff_zero_intensity float 0.4 For the intensity based drib-off, the probability of each point with zero intensity being dropped.
sensor_tick float 0.0 Simulation seconds betwixt sensor captures (ticks).
noise_stddev float 0.0 Standard deviation of the dissonance model to disturb each bespeak forth the vector of its raycast.

Output attributes

Sensor information attribute Type Description
frame int Frame number when the measurement took identify.
timestamp double Simulation time of the measurement in seconds since the beginning of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the fourth dimension of the measurement.
horizontal_angle float Angle (radians) in the XY plane of the LIDAR in the current frame.
channels int Number of channels (lasers) of the LIDAR.
get_point_count(channel) int Number of points per channel captured this frame.
raw_data bytes Array of 32-bits floats (XYZI of each indicate).

Obstruction detector

  • Pattern: sensor.other.obstacle
  • Output: carla.ObstacleDetectionEvent per obstruction (unless sensor_tick says otherwise).

Registers an effect every fourth dimension the parent actor has an obstacle ahead. In club to anticipate obstacles, the sensor creates a capsular shape ahead of the parent vehicle and uses it to cheque for collisions. To ensure that collisions with whatsoever kind of object are detected, the server creates "fake" actors for elements such every bit buildings or bushes and so the semantic tag tin can be retrieved to identify it.

Blueprint attribute Type Default Description
distance float 5 Altitude to trace.
hit_radius float 0.5 Radius of the trace.
only_dynamics bool False If true, the trace will only consider dynamic objects.
debug_linetrace bool False If truthful, the trace volition be visible.
sensor_tick float 0.0 Simulation seconds betwixt sensor captures (ticks).

Output attributes

Sensor data attribute Type Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the beginning of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the fourth dimension of the measurement.
actor carla.Role player Player that detected the obstacle (parent role player).
other_actor carla.Histrion Actor detected as an obstacle.
distance bladder Distance from actor to other_actor.

Radar sensor

  • Blueprint: sensor.other.radar
  • Output: carla.RadarMeasurement per step (unless sensor_tick says otherwise).

The sensor creates a conic view that is translated to a 2D indicate map of the elements in sight and their speed regarding the sensor. This can be used to shape elements and evaluate their move and direction. Due to the use of polar coordinates, the points will concentrate around the center of the view.

Points measured are contained in carla.RadarMeasurement as an assortment of carla.RadarDetection, which specifies their polar coordinates, distance and velocity. This raw information provided by the radar sensor tin be hands converted to a format manageable by numpy:

          # To get a numpy [[vel, azimuth, altitude, depth],...[,,,]]: points = np.frombuffer(radar_data.raw_data, dtype=np.dtype('f4')) points = np.reshape(points, (len(radar_data), iv))

The provided script manual_control.py uses this sensor to show the points being detected and paint them white when static, cherry-red when moving towards the object and blue when moving away:

ImageRadar

Blueprint attribute Type Default Description
horizontal_fov float 30.0 Horizontal field of view in degrees.
points_per_second int 1500 Points generated by all lasers per second.
range float 100 Maximum distance to measure/raycast in meters.
sensor_tick float 0.0 Simulation seconds between sensor captures (ticks).
vertical_fov float xxx.0 Vertical field of view in degrees.

Output attributes

Sensor data attribute Type Description
raw_data carla.RadarDetection The list of points detected.
RadarDetection attributes Blazon Clarification
altitude float Altitude angle in radians.
azimuth float Azimuth angle in radians.
depth float Distance in meters.
velocity float Velocity towards the sensor.

RGB photographic camera

  • Blueprint: sensor.camera.rgb
  • Output: carla.Image per step (unless sensor_tick says otherwise)..

The "RGB" camera acts as a regular camera capturing images from the scene. carla.colorConverter

If enable_postprocess_effects is enabled, a set of post-process effects is applied to the image for the sake of realism:

  • Vignette: darkens the border of the screen.
  • Grain jitter: adds some noise to the render.
  • Blossom: intense lights fire the area around them.
  • Auto exposure: modifies the prototype gamma to simulate the middle adaptation to darker or brighter areas.
  • Lens flares: simulates the reflection of bright objects on the lens.
  • Depth of field: blurs objects well-nigh or very far abroad of the camera.

The sensor_tick tells how fast we desire the sensor to capture the data. A value of i.5 means that we want the sensor to capture information each 2d and a half. Past default a value of 0.0 means as fast as possible.

ImageRGB

Basic photographic camera attributes

Blueprint attribute Type Default Description
bloom_intensity bladder 0.675 Intensity for the blossom post-process result, 0.0 for disabling it.
fov float 90.0 Horizontal field of view in degrees.
fstop float 1.4 Opening of the photographic camera lens. Aperture is ane/fstop with typical lens going down to f/ane.2 (larger opening). Larger numbers will reduce the Depth of Field result.
image_size_x int 800 Prototype width in pixels.
image_size_y int 600 Image height in pixels.
iso bladder 100.0 The camera sensor sensitivity.
gamma float 2.ii Target gamma value of the camera.
lens_flare_intensity float 0.1 Intensity for the lens flare post-process effect, 0.0 for disabling it.
sensor_tick float 0.0 Simulation seconds betwixt sensor captures (ticks).
shutter_speed float 200.0 The camera shutter speed in seconds (ane.0/s).

Camera lens distortion attributes

Pattern attribute Type Default Description
lens_circle_falloff float 5.0 Range: [0.0, 10.0]
lens_circle_multiplier float 0.0 Range: [0.0, 10.0]
lens_k float -ane.0 Range: [-inf, inf]
lens_kcube float 0.0 Range: [-inf, inf]
lens_x_size float 0.08 Range: [0.0, 1.0]
lens_y_size float 0.08 Range: [0.0, i.0]

Advanced camera attributes

Since these effects are provided past UE, please brand sure to check their documentation:

  • Automatic Exposure
  • Cinematic Depth of Field Method
  • Color Grading and Filmic Tonemapper
Blueprint attribute Type Default Description
min_fstop float i.2 Maximum aperture.
blade_count int v Number of blades that brand up the diaphragm mechanism.
exposure_mode str histogram Can be manual or histogram. More in UE4 docs.
exposure_compensation bladder Linux: +0.75
Windows: 0.0		 Logarithmic adjustment for the exposure. 0: no aligning, -ane:2x darker, -2:4 darker, 1:2x brighter, 2:4x brighter.
exposure_min_bright float x.0 In exposure_mode: "histogram". Minimum effulgence for auto exposure. The lowest the eye tin can adapt within. Must be greater than 0 and less than or equal to exposure_max_bright.
exposure_max_bright float 12.0 In `exposure_mode: "histogram"`. Maximum brightness for auto exposure. The highestthe eye can adapt within. Must be greater than 0 and greater than or equal to `exposure_min_bright`.
exposure_speed_up bladder 3.0 In exposure_mode: "histogram". Speed at which the adaptation occurs from night to bright environment.
exposure_speed_down float 1.0 In exposure_mode: "histogram". Speed at which the adaptation occurs from bright to dark environment.
calibration_constant float xvi.0 Calibration constant for 18% albedo.
focal_distance bladder thousand.0 Distance at which the depth of field effect should be sharp. Measured in cm (UE units).
blur_amount float 1.0 Strength/intensity of move blur.
blur_radius float 0.0 Radius in pixels at 1080p resolution to emulate atmospheric scattering co-ordinate to distance from camera.
motion_blur_intensity float 0.45 Strength of motion blur [0,1].
motion_blur_max_distortion bladder 0.35 Max baloney caused by motion blur. Percentage of screen width.
motion_blur_min_object_screen_size float 0.one Percentage of screen width objects must have for motion blur, lower value means less draw calls.
gradient float 0.88 Steepness of the Due south-curve for the tonemapper. Larger values make the slope steeper (darker) [0.0, one.0].
toe float 0.55 Adjusts nighttime color in the tonemapper [0.0, i.0].
shoulder float 0.26 Adjusts bright color in the tonemapper [0.0, one.0].
black_clip bladder 0.0 This should NOT be adjusted. Sets where the crossover happens and black tones showtime to cutting off their value [0.0, ane.0].
white_clip float 0.04 Set where the crossover happens and white tones start to cut off their value. Subtle alter in well-nigh cases [0.0, 1.0].
temp float 6500.0 White balance in relation to the temperature of the light in the scene. White lite: when this matches lite temperature. Warm lite: When higher than the lite in the scene, it is a yellowish color. Absurd lite: When lower than the low-cal. Blueish color.
tint bladder 0.0 White balance temperature tint. Adjusts cyan and magenta colour ranges. This should be used along with the white balance Temp property to become accurate colors. Nether some calorie-free temperatures, the colors may announced to be more yellow or blue. This can be used to balance the resulting color to look more than natural.
chromatic_aberration_intensity float 0.0 Scaling factor to control colour shifting, more noticeable on the screen borders.
chromatic_aberration_offset float 0.0 Normalized distance to the middle of the paradigm where the consequence takes place.
enable_postprocess_effects bool True Post-procedure effects activation.

Output attributes

Sensor data attribute Type Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the beginning of the episode.
transform carla.Transform Location and rotation in earth coordinates of the sensor at the time of the measurement.
width int Prototype width in pixels.
tiptop int Image tiptop in pixels.
fov float Horizontal field of view in degrees.
raw_data bytes Array of BGRA 32-flake pixels.
  • Pattern: sensor.other.rss
  • Output: carla.RssResponse per step (unless sensor_tick says otherwise).

Of import

Information technology is highly recommended to read the specific rss documentation before reading this.

This sensor integrates the C++ Library for Responsibleness Sensitive Safety in CARLA. It is disabled by default in CARLA, and it has to be explicitly built in lodge to be used.

The RSS sensor calculates the RSS state of a vehicle and retrieves the current RSS Response as sensor information. The carla.RssRestrictor will apply this data to adapt a carla.VehicleControl before applying it to a vehicle.

These controllers tin can exist generated by an Automated Driving stack or user input. For instance, hereunder there is a fragment of code from PythonAPI/examples/rss/manual_control_rss.py, where the user input is modified using RSS when necessary.

ane. Checks if the RssSensor generates a valid response containing restrictions. two. Gathers the current dynamics of the vehicle and the vehicle physics. 3. Applies restrictions to the vehicle command using the response from the RssSensor, and the electric current dynamics and physicis of the vehicle.

          rss_proper_response = self._world.rss_sensor.proper_response if cocky._world.rss_sensor and cocky._world.rss_sensor.response_valid else None if rss_proper_response: ...         vehicle_control = cocky._restrictor.restrict_vehicle_control(             vehicle_control, rss_proper_response, self._world.rss_sensor.ego_dynamics_on_route, cocky._vehicle_physics)

The blueprint for this sensor has no modifiable attributes. Yet, the carla.RssSensor object that it instantiates has attributes and methods that are detailed in the Python API reference. Here is a summary of them.

carla.RssSensor variables Type Clarification
ego_vehicle_dynamics ad.rss.world.RssDynamics RSS parameters to be applied for the ego vehicle
other_vehicle_dynamics advertisement.rss.earth.RssDynamics RSS parameters to exist applied for the other vehicles
pedestrian_dynamics advert.rss.world.RssDynamics RSS parameters to be applied for pedestrians
road_boundaries_mode carla.RssRoadBoundariesMode Enables/Disables the stay on road feature. Default is Off. 
          # Fragment of rss_sensor.py # The carla.RssSensor is updated when listening for a new carla.RssResponse def _on_rss_response(weak_self, response): ...         self.timestamp = response.timestamp         self.response_valid = response.response_valid         self.proper_response = response.proper_response         self.ego_dynamics_on_route = response.ego_dynamics_on_route         self.rss_state_snapshot = response.rss_state_snapshot         self.situation_snapshot = response.situation_snapshot         self.world_model = response.world_model

Warning

This sensor works fully on the client side. There is no blueprint in the server. Changes on the attributes will have effect subsequently the listen() has been called.

The methods bachelor in this class are related to the routing of the vehicle. RSS calculations are always based on a road of the ego vehicle through the road network.

The sensor allows to command the considered route by providing some central points, which could be the carla.Transform in a carla.Waypoint. These points are best selected later on the intersections to force the road to take the desired turn.

carla.RssSensor methods Description
routing_targets Get the current list of routing targets used for route.
append_routing_target Append an additional position to the current routing targets.
reset_routing_targets Deletes the appended routing targets.
drop_route Discards the current road and creates a new ane.
register_actor_constellation_callback Register a callback to customize the calculations.
set_log_level Sets the log level.
set_map_log_level Sets the log level used for map related logs. 
          # Update the current route self.sensor.reset_routing_targets() if routing_targets:     for target in routing_targets:         self.sensor.append_routing_target(target)

Annotation

If no routing targets are divers, a random route is created.

Output attributes

carla.RssResponse attributes Blazon Description
response_valid bool Validity of the response information.
proper_response advertisement.rss.land.ProperResponse Proper response that the RSS calculated for the vehicle including acceleration restrictions.
rss_state_snapshot ad.rss.state.RssStateSnapshot RSS states at the current point in time. This is the detailed individual output of the RSS calclulations.
situation_snapshot ad.rss.state of affairs.SituationSnapshot RSS state of affairs at the current point in time. This is the processed input information for the RSS calclulations.
world_model advertisement.rss.world.WorldModel RSS earth model at the current point in fourth dimension. This is the input information for the RSS calculations.
ego_dynamics_on_route carla.RssEgoDynamicsOnRoute Current ego vehicle dynamics regarding the route.

In example a actor_constellation_callback is registered, a telephone call is triggered for:

  1. default calculation (actor_constellation_data.other_actor=None)
  2. per-actor calculation
          # Fragment of rss_sensor.py # The function is registered as actor_constellation_callback def _on_actor_constellation_request(self, actor_constellation_data):     actor_constellation_result = carla.RssActorConstellationResult()     actor_constellation_result.rss_calculation_mode = ad.rss.map.RssMode.NotRelevant     actor_constellation_result.restrict_speed_limit_mode = ad.rss.map.RssSceneCreation.RestrictSpeedLimitMode.IncreasedSpeedLimit10     actor_constellation_result.ego_vehicle_dynamics = self.current_vehicle_parameters     actor_constellation_result.actor_object_type = advertising.rss.world.ObjectType.Invalid     actor_constellation_result.actor_dynamics = cocky.current_vehicle_parameters      actor_id = -1     actor_type_id = "none"     if actor_constellation_data.other_actor != None:         # customize actor_constellation_result for specific actor         ...     else:         # default         ...     render actor_constellation_result

Semantic LIDAR sensor

  • Blueprint: sensor.lidar.ray_cast_semantic
  • Output: carla.SemanticLidarMeasurement per step (unless sensor_tick says otherwise).

This sensor simulates a rotating LIDAR implemented using ray-casting that exposes all the information about the raycast hit. Its behaviour is quite similar to the LIDAR sensor, but in that location are 2 principal differences between them.

  • The raw data retrieved by the semantic LIDAR includes more data per point.
    • Coordinates of the indicate (as the normal LIDAR does).
    • The cosine between the angle of incidence and the normal of the surface hit.
    • Example and semantic ground-truth. Basically the index of the CARLA object hitting, and its semantic tag.
  • The semantic LIDAR does not include neither intensity, drop-off nor noise model attributes.

The points are computed by adding a laser for each channel distributed in the vertical FOV. The rotation is simulated computing the horizontal angle that the LIDAR rotated in a frame. The betoken deject is calculated by doing a ray-cast for each light amplification by stimulated emission of radiation in every footstep.

          points_per_channel_each_step = points_per_second / (FPS * channels)

A LIDAR measurement contains a package with all the points generated during a 1/FPS interval. During this interval the physics are not updated so all the points in a measurement reflect the aforementioned "static picture show" of the scene.

This output contains a cloud of lidar semantic detections and therefore, information technology tin be iterated to remember a list of their carla.SemanticLidarDetection:

          for detection in semantic_lidar_measurement:     print(detection)

The rotation of the LIDAR can be tuned to comprehend a specific angle on every simulation step (using a fixed time-step). For case, to rotate once per step (full circle output, every bit in the pic beneath), the rotation frequency and the false FPS should be equal.
1. Set the sensor's frequency sensors_bp['lidar'][0].set_attribute('rotation_frequency','ten').
2. Run the simulation using python3 config.py --fps=x.

LidarPointCloud

SemanticLidar attributes

Blueprint attribute Type Default Clarification
channels int 32 Number of lasers.
range float x.0 Maximum altitude to measure/raycast in meters (centimeters for CARLA 0.nine.6 or previous).
points_per_second int 56000 Points generated by all lasers per 2nd.
rotation_frequency float x.0 LIDAR rotation frequency.
upper_fov float x.0 Bending in degrees of the highest laser.
lower_fov bladder -30.0 Angle in degrees of the lowest light amplification by stimulated emission of radiation.
horizontal_fov float 360.0 Horizontal field of view in degrees, 0 - 360.
sensor_tick float 0.0 Simulation seconds between sensor captures (ticks).

Output attributes

Sensor data aspect Blazon Description
frame int Frame number when the measurement took place.
timestamp double Simulation time of the measurement in seconds since the beginning of the episode.
transform carla.Transform Location and rotation in earth coordinates of the sensor at the time of the measurement.
horizontal_angle float Bending (radians) in the XY plane of the LIDAR in the current frame.
channels int Number of channels (lasers) of the LIDAR.
get_point_count(aqueduct) int Number of points per channel captured in the electric current frame.
raw_data bytes Array containing the bespeak cloud with case and semantic information. For each point, four 32-$.25 floats are stored.
XYZ coordinates.
cosine of the incident bending.
Unsigned int containing the index of the object hit.
Unsigned int containing the semantic tag of the object it.

Semantic segmentation camera

  • Pattern: sensor.camera.semantic_segmentation
  • Output: carla.Image per step (unless sensor_tick says otherwise).

This photographic camera classifies every object in sight by displaying information technology in a dissimilar color according to its tags (eastward.g., pedestrians in a different color than vehicles). When the simulation starts, every element in scene is created with a tag. And then it happens when an actor is spawned. The objects are classified by their relative file path in the project. For example, meshes stored in Unreal/CarlaUE4/Content/Static/Pedestrians are tagged as Pedestrian.

ImageSemanticSegmentation

The server provides an image with the tag data encoded in the ruby-red channel: A pixel with a carmine value of 10 belongs to an object with tag x. This raw carla.Image tin can be stored and converted it with the aid of CityScapesPalette in carla.ColorConverter to utilize the tags information and show picture with the semantic division.

          ... raw_image.save_to_disk("path/to/save/converted/image",carla.cityScapesPalette)

The post-obit tags are currently available:

Value Tag Converted color Description
0 Unlabeled (0, 0, 0) Elements that have not been categorized are considered Unlabeled. This category is meant to be empty or at least contain elements with no collisions.
one Building (70, 70, seventy) Buildings like houses, skyscrapers,... and the elements fastened to them.
E.1000. air conditioners, scaffolding, awning or ladders and much more.
2 Fence (100, 40, twoscore) Barriers, railing, or other upright structures. Basically forest or wire assemblies that enclose an area of ground.
3 Other (55, 90, lxxx) Everything that does non vest to any other category.
4 Pedestrian (220, 20, sixty) Humans that walk or ride/bulldoze any kind of vehicle or mobility system.
E.g. bicycles or scooters, skateboards, horses, roller-blades, wheel-chairs, etc.
five Pole (153, 153, 153) Pocket-size mainly vertically oriented pole. If the pole has a horizontal part (often for traffic calorie-free poles) this is also considered pole.
E.1000. sign pole, traffic light poles.
6 RoadLine (157, 234, l) The markings on the route.
7 Route (128, 64, 128) Role of ground on which cars ordinarily drive.
E.g. lanes in any directions, and streets.
8 SideWalk (244, 35, 232) Part of ground designated for pedestrians or cyclists. Delimited from the road by some obstacle (such equally curbs or poles), non only by markings. This label includes a possibly delimiting curb, traffic islands (the walkable part), and pedestrian zones.
9 Vegetation (107, 142, 35) Trees, hedges, all kinds of vertical vegetation. Ground-level vegetation is considered Terrain.
10 Vehicles (0, 0, 142) Cars, vans, trucks, motorcycles, bikes, buses, trains.
11 Wall (102, 102, 156) Private continuing walls. Not function of a building.
12 TrafficSign (220, 220, 0) Signs installed by the state/city authorization, unremarkably for traffic regulation. This category does non include the poles where signs are attached to.
E.g. traffic- signs, parking signs, management signs...
13 Heaven (seventy, 130, 180) Open heaven. Includes clouds and the sun.
14 Basis (81, 0, 81) Any horizontal basis-level structures that does non match any other category. For instance areas shared by vehicles and pedestrians, or flat roundabouts delimited from the road past a curb.
15 Bridge (150, 100, 100) But the structure of the bridge. Fences, people, vehicles, an other elements on top of it are labeled separately.
sixteen RailTrack (230, 150, 140) All kind of rail tracks that are non-drivable past cars.
Eastward.one thousand. subway and train runway tracks.
17 GuardRail (180, 165, 180) All types of baby-sit rails/crash barriers.
18 TrafficLight (250, 170, 30) Traffic light boxes without their poles.
19 Static (110, 190, 160) Elements in the scene and props that are immovable.
E.g. burn down hydrants, fixed benches, fountains, bus stops, etc.
20 Dynamic (170, 120, l) Elements whose position is susceptible to alter over time.
E.g. Movable trash bins, buggies, numberless, wheelchairs, animals, etc.
21 Water (45, 60, 150) Horizontal water surfaces.
E.g. Lakes, sea, rivers.
22 Terrain (145, 170, 100) Grass, basis-level vegetation, soil or sand. These areas are not meant to be driven on. This label includes a perchance delimiting curb.

Note

Read this tutorial to create new semantic tags.

Bones camera attributes

Blueprint aspect Type Default Description
fov bladder ninety.0 Horizontal field of view in degrees.
image_size_x int 800 Image width in pixels.
image_size_y int 600 Image acme in pixels.
sensor_tick bladder 0.0 Simulation seconds between sensor captures (ticks).

Camera lens distortion attributes

Blueprint attribute Blazon Default Description
lens_circle_falloff float 5.0 Range: [0.0, 10.0]
lens_circle_multiplier bladder 0.0 Range: [0.0, 10.0]
lens_k bladder -ane.0 Range: [-inf, inf]
lens_kcube float 0.0 Range: [-inf, inf]
lens_x_size float 0.08 Range: [0.0, i.0]
lens_y_size float 0.08 Range: [0.0, i.0]

Output attributes

Sensor data aspect Type Description
fov bladder Horizontal field of view in degrees.
frame int Frame number when the measurement took place.
height int Image acme in pixels.
raw_data bytes Array of BGRA 32-fleck pixels.
timestamp double Simulation time of the measurement in seconds since the get-go of the episode.
transform carla.Transform Location and rotation in globe coordinates of the sensor at the time of the measurement.
width int Image width in pixels.

DVS camera

  • Blueprint: sensor.camera.dvs
  • Output: carla.DVSEventArray per step (unless sensor_tick says otherwise).

A Dynamic Vision Sensor (DVS) or Event camera is a sensor that works radically differently from a conventional photographic camera. Instead of capturing intensity images at a stock-still rate, event cameras mensurate changes of intensity asynchronously, in the grade of a stream of events, which encode per-pixel effulgence changes. Event cameras possess distinct properties when compared to standard cameras. They have a very high dynamic range (140 dB versus 60 dB), no motility blur, and loftier temporal resolution (in the lodge of microseconds). Upshot cameras are thus sensors that can provide loftier-quality visual information fifty-fifty in challenging loftier-speed scenarios and loftier dynamic range environments, enabling new awarding domains for vision-based algorithms.

The DVS camera outputs a stream of events. An issue e=(x,y,t,pol) is triggered at a pixel x, y at a timestamp t when the change in logarithmic intensity 50 reaches a predefined abiding threshold C (typically between xv% and 30%).

L(x,y,t) - L(ten,y,t-\delta t) = politico C

t-\delta t is the time when the last result at that pixel was triggered and politician is the polarity of the event according to the sign of the brightness change. The polarity is positive +1 when there is increase in brightness and negative -one when a decrement in brightness occurs. The working principles depicted in the following figure. The standard camera outputs frames at a fixed rate, thus sending redundant information when no motility is nowadays in the scene. In contrast, result cameras are data-driven sensors that answer to brightness changes with microsecond latency. At the plot, a positive (resp. negative) event (bluish dot, resp. red dot) is generated whenever the (signed) brightness change exceeds the dissimilarity threshold C for one dimension x over time t. Observe how the event rate grows when the indicate changes rapidly.

DVSCameraWorkingPrinciple

The current implementation of the DVS camera works in a uniform sampling manner between two sequent synchronous frames. Therefore, in order to emulate the high temporal resolution (society of microseconds) of a existent effect camera, the sensor requires to execute at a loftier frequency (much higher frequency than a conventional photographic camera). Effectively, the number of events increases the faster a CARLA motorcar drives. Therefore, the sensor frequency should increase accordingly with the dynamics of the scene. The user should observe a balance between fourth dimension accurateness and computational toll.

The provided script manual_control.py uses the DVS camera in order to show how to configure the sensor, how to go the stream of events and how to draw such events in an paradigm format, unremarkably chosen event frame.

Note that due to the sampling method of the DVS photographic camera, if there is no pixel difference between two consecutive synchronous frames the camera will non render an prototype. This will always occur in the start frame, as there is no previous frame to compare to and as well in the result that there has been no movement betwixt frames.

DVSCameraWorkingPrinciple

DVS is a camera and therefore has all the attributes available in the RGB camera. Nevertheless, there are few attributes sectional to the working principle of an Issue camera.

DVS camera attributes

Design aspect Type Default Description
positive_threshold float 0.three Positive threshold C associated to a increment in brightness change (0-one).
negative_threshold float 0.3 Negative threshold C associated to a decrement in brightness change (0-1).
sigma_positive_threshold float 0 White dissonance standard deviation for positive events (0-1).
sigma_negative_threshold float 0 White noise standard deviation for negative events (0-1).
refractory_period_ns int 0.0 Refractory period (fourth dimension during which a pixel cannot fire events just after information technology fired one), in nanoseconds. It limits the highest frequency of triggering events.
use_log bool true Whether to work in the logarithmic intensity scale.
log_eps float 0.001 Epsilon value used to convert images to log: L = log(eps + I / 255.0).
Where I is the grayscale value of the RGB paradigm:
I = 0.2989*R + 0.5870*Thou + 0.1140*B.

Optical Flow Camera

The Optical Flow photographic camera captures the motility perceived from the point of view of the camera. Every pixel recorded past this sensor encodes the velocity of that signal projected to the prototype plane. The velocity of a pixel is encoded in the range [-2,2]. To obtain the motility in pixel units, this information tin be scaled with the paradigm size to [-2 * image_size, ii * image_size].

optical_flow

Optical Flow camera attributes

Pattern attribute Blazon Default Clarification
image_size_x int 800 Image width in pixels.
image_size_y int 600 Image acme in pixels.
fov bladder 90.0 Horizontal field of view in degrees.
sensor_tick float 0.0 Simulation seconds between sensor captures (ticks).

Optical Menses camera lens distortion attributes

Blueprint attribute Type Default Description
lens_circle_falloff float 5.0 Range: [0.0, 10.0]
lens_circle_multiplier float 0.0 Range: [0.0, ten.0]
lens_k float -1.0 Range: [-inf, inf]
lens_kcube bladder 0.0 Range: [-inf, inf]
lens_x_size float 0.08 Range: [0.0, ane.0]
lens_y_size float 0.08 Range: [0.0, ane.0]

Output attributes

Sensor data attribute Type Description
frame int Frame number when the measurement took identify.
timestamp double Simulation fourth dimension of the measurement in seconds since the offset of the episode.
transform carla.Transform Location and rotation in world coordinates of the sensor at the time of the measurement.
width int Epitome width in pixels.
height int Image height in pixels.
fov float Horizontal field of view in degrees.
raw_data bytes Assortment of BGRA 64-fleck pixels containing two float values.