Questions tagged [avro]
Apache Avro is a data serialization framework primarily used in Apache Hadoop.
avro
3,755
questions
1
vote
1
answer
56
views
Got error "Serializer failed" while serializing to AVRO with a schema including optional fields
I am producing data to Kafka in AVRO format.
I created a schema by
curl --location 'https://confluent-schema-registry.example.com/subjects/production.iot.device.avro-value/versions' \
--header '...
- 48.8k
0
votes
0
answers
12
views
spark3 submit job is unable to read metadata of Hive avro table
I can describe hive table in spark3-shell, spark-shell, beeline and spark-submit job properly. But when I describe the table in spark3-submit job it gives this error:
Unable to get field from serde: ...
1
vote
0
answers
23
views
How do I consume AVRO messages from a kafka topic in go? Can I ignore the first 5 bytes and hardcode the schema?
I have to update a service written in go to consume messages from a new avro kafka topic, before it was just JSON. I originally wrote the code pretty much following the documentation of hamba/avro ...
- 11
0
votes
1
answer
43
views
Avro: Decode using multiprocessing in Python
In terms of best performance with less latency, which asynchronous approach would be better towards decoding the Avro message that is consumed from the Kafka topic?
I was using concurrent futures with ...
- 129
-2
votes
0
answers
15
views
How to prevent nullable fields in Avro IDL to complicate JSON message in Kafka Topic
I have a reasonable complex object hierarchy which I constructed in IDL (just a very small part):
record Action {
Location location;
GeneralInformation? generalInformation=null
}
The generated ...
- 9,944
0
votes
1
answer
35
views
Insert JSON into Postgres via NiFi
I'm trying insert record with JSON into Postgres table with column type of jsonb using AVROSchemaRegistry and Processor "PutDatabaseRecord".
Apache NiFi version: nifi-2.0.0-M2-RC4
Postgres ...
1
vote
0
answers
20
views
Print List of Generic Interface in Java 11
I have this property in a class:
private final List<KafkaInterface<EventWhateverDTO>> producers;
Code of interface:
public interface KafkaInterface<T> {
void produceEvent(Map<...
- 1,599
0
votes
0
answers
25
views
Output collections -> Elements estimated added/size estimation dataflow gcp don't shown
I'm using the JDK 17 and Apache Beam 2.50 and on the gcp dataflow graph and on the read from Avro/write to BigQuery I get the
READ RAW+ MESSAGES FROM AVRO file/Read.out0
Elements added : 332
Estimated ...
0
votes
0
answers
24
views
org.apache.kafka.common.errors.SerializationException: Error registering Avro schema
https://i.sstatic.net/jtpBxVPF.png
I send a message from procuder to kafkaya and consumer listens to it.but i can't save the schema in shema registry
2024-07-26T10:22:03.461+03:00 INFO 19888 --- [nio-...
0
votes
0
answers
53
views
Failing spring boot tests with TestChannelBinderConfiguration after spring boot 3.3.2 update
In my project I updated the spring-boot-parent dependency from 3.3.1 to 3.3.2, and my tests began to fail. The bigger issue is, that not all of them, and not always. Approximately 50%-50%, whether the ...
1
vote
1
answer
30
views
Scalable AVRO schema for Kafka topic
I have an application in which I am using Kafka with a schema registry to produce messages. I have a topic called "plans" which contains fields like id, name, and type (currently having ...
-2
votes
1
answer
21
views
dead-letter publication using bytes in schama, I need avro
I have some problem-I have application in SpringBoot and I am using Kafka with retries. I have producer who avro compatibiled message to my topic and I have consumer to consume this message. It works ...
- 1
0
votes
0
answers
28
views
How to append time-series data with PyArrow Datasets?
Problem
I'm looking to store time-series data that's being aggregated live to Parquet Datasets via PyArrow. I receive live batched data, for example, video view count each hour for the last 24 hours. ...
- 325
0
votes
0
answers
21
views
How to write List<Object> in avro?
I have some problem. I need to create avro schema for Kafka. In one of my java class i have field with list of Object:
public class Model {
@XmlAnyElement(lax = true)
protected List<Object&...
- 1
1
vote
1
answer
61
views
Convert avro file to json with powershell
I'm capturing data in avro via Azure EventHub so it can be handled via a Azure Automation Runbook. The issue I'm facing is when I try to parse the avro file to be used in a powershell script.
Here the ...
