Setting onClickListeners for buttons in scenes programmatically

I have 2 layouts which contain the same buttons

layout_1.xml

  <RelativeLayout
        android:layout_width="wrap_content"
        android:layout_height="wrap_content">
    <Button
        android:id="@+id/button_1"
        android:text="button2"
        android:background="@android:color/black"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content" />
    </RelativeLayout>

and layout_2.xml

<RelativeLayout
        android:layout_width="wrap_content"
        android:layout_height="wrap_content">
    <Button
        android:id="@+id/button_1"
        android:text="button2"
        android:background="@android:color/white"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content" />
    </RelativeLayout>

Please assume these are all valid layouts etc.(I am just adding the relevant code.).

So in my fragment ,I inflate and use layout_1.xml in onCreateView.I want to toggle between the 2 scenes using button_1. I can set the listener for button_1 in layout_1.xml during the onCreateView(). The problem is trying to set a listener on that button in the second view.i.e. the listener does not activate for the second scene(with layout_2.xml).And hence i canot toggle between the 2 scenes.Is there a way to achieve this?