Outputting data recieved from an android device as sound

The goal: I've got a small android app running, which accepts input via the mic and streams it back in realtime - I'm trying to modify it so that it streams the data over to a java server on my PC.

The problem: All I get is a "tappy" noise over and over again, or a continous "beep" sound (The two noises change depending on how I'm reading the data serverside). I realize it's most likely something to do with the way I'm playing back the sound, since on the android device it's able to play its own sound accurately.

The code: On both the client and server side there is only one class.

How the server is receiving and trying to play the data:

public static void main(String[] args) {
    try {
        ServerSocket x = new ServerSocket(6790);
        try {
            while (true) {
                System.out.println("started");

                Socket clientSocket = x.accept();

                System.out.println("Connected" + clientSocket.getInetAddress());
                InputStream y = clientSocket.getInputStream();
                // ObjectInputStream in = new ObjectInputStream(y);

                AudioFormat format = new AudioFormat(8000, 16, 1, true, true);

                AudioInputStream audIn = new AudioInputStream(y, format, 1);
                SourceDataLine speakers;
                try {
                    byte[] data = new byte[32];
                    int numBytesRead;
                    int CHUNK_SIZE = 1024;
                    int bytesRead = 0;
                    DataLine.Info dataLineInfo = new DataLine.Info(SourceDataLine.class, format);
                    speakers = (SourceDataLine) AudioSystem.getLine(dataLineInfo);
                    speakers.open(format);
                    speakers.start();
                    while (bytesRead < 100000) {
                        numBytesRead = audIn.read(data, 0, 16); // I tried y.read(data, 0, 16); but it doesn't work
                        bytesRead += numBytesRead;
                        //for immediate playback
                        speakers.write(data, 0, 16);
                    }
                    speakers.drain();
                    speakers.close();
                } catch (LineUnavailableException e) {
                    e.printStackTrace();
                }    
            }
        } catch (Exception e1) {
            e1.printStackTrace();
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

How I'm recording and sending from the client side:

public void run() {
    try {
        Log.i("Audio", "Running Audio Thread");
        AudioRecord recorder = null;
        AudioTrack track = null;
        ObjectOutputStream out = new ObjectOutputStream(x.getOutputStream());
        short[][] buffers = new short[256][160];
        int ix = 0;

        /*
         * Initialize buffer to hold continuously recorded audio data, start recording, and start
         * playback.
         */

        int N = AudioRecord.getMinBufferSize(8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
        recorder = new AudioRecord(AudioSource.MIC, 8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, N * 1);
        track = new AudioTrack(AudioManager.STREAM_MUSIC, 8000, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, N * 1,
                AudioTrack.MODE_STREAM);
        recorder.startRecording();
        track.play();

        /*
         * Loops until something outside of this thread stops it.
         * Reads the data from the recorder and writes it to the audio track for playback.
         */
        while (!stopped) {
            Log.i("Map", "Writing new data to buffer");
            short[] buffer = buffers[ix++ % buffers.length];
            N = recorder.read(buffer, 0, buffer.length);
            track.write(buffer, 0, buffer.length);
            out.writeObject(buffer);
            // out.flush();
        }

        /*
        * Frees the thread's resources after the loop completes so that it can be run again
        */
        recorder.stop();
        recorder.release();
        track.stop();
        track.release();
        Log.d("Exit", "Exiting thread");
    } catch (Throwable x) {
        Log.w("Audio", "Error reading voice audio", x);
    }

}

I know full well that UDP is usually what's used for voice streaming, but this is the only set of code that even comes close to working. In previous versions, I've tried using things like RTP and WebRTC, as well as multiple libraries, this is the only version where I've been able to get a "live" playback.

I've already tried a ton of different approaches, such as using Clip and ObjectInputStream, but nothing seems to be working. In this version, I am sending short[] via ObjectOutputStream, and reading via AudioInputStream, but I'm open to sending and reading any kind of data as long as it works. I've tried converting the short[] to a byte[] and playing it back, but it didn't seem to help.

Errors:

No actual errors are to be reported (as in no exceptions were thrown). There is also nothing wrong in the LogCat - note that the only working configurations for format are:

AudioFormat format = new AudioFormat(8000, 16, 1, true, true);  //louder
AudioFormat format = new AudioFormat(8000, 16, 1, true, false); //quieter

The other 2 combinations of signed/unsigned + endian throw exceptions.

I can also confirm that the data is actually being sent through the connection - I used wireshark to tap into it and I can see clearly the source and destination, and data being sent. I'm sending to the correct port for sure.

If any more code is needed, ask and I will edit and put it up.

TLDR; How can I play back sound sent from an android device to a computer running a java server?